C code has certain limitations in the REXLANG block. A detailed description of the block is available here: https://www.rexygen.com/doc/ENGLISH/MANUALS/BRef/REXLANG.html#x274-27300015.
Examples for using the REXLANG block are available in the installation. By default on the C:\Program Files\REX Controls\REXYGEN 126.96.36.19991\Examples\0203_Function_Blocks_REXLANG path.
Compilation takes place only when translating the whole project in Rexygen Studio.
Unfortunately, .so or .dll files are not supported in REXLANG .
In REXYGEN, it is possible to call standard C language scripts with linked .so or .dll files. Scripts can be executed using the EPC block (documentation here).
If you can use Python instead of C, take a look at the Python block (documentation here).