# How to create and debug Ruby gem with C (native) extension This article explains how to create a minimum Ruby gem with native extension written in C and debug it with gdb. I uploaded complete code to [GitHub](https://github.com/wataash/ruby-example-c-ext). Tested with: - Ubuntu 20.04 (with self-built Linux kernel 5.11.22+) - Ruby 2.7.4 {{< admonition warning >}} **Using Ruby version 2.7.4 is strongly recommended at the first time of doing this tutorial.** I think making Ruby-C-extensions has a high probability of encountering errors -- for instance, as of 2021-08-14, with Ruby 3.0.2, `bundle exec rake install` fails with the following error: ```log $ bundle exec rake install install -c tmp/x86_64-linux/example_ext/3.0.2/example_ext.so lib/example_ext/example_ext.so cp tmp/x86_64-linux/example_ext/3.0.2/example_ext.so tmp/x86_64-linux/stage/lib/example_ext/example_ext.so example_ext 0.1.0 built to pkg/example_ext-0.1.0.gem. rake aborted! Couldn't install gem, run `gem install /tmp/ruby-c-extension/example_ext/pkg/example_ext-0.1.0.gem' for more detailed output /home/wsh/.rbenv/versions/3.0.2-dbg/bin/bundle:23:in `load' /home/wsh/.rbenv/versions/3.0.2-dbg/bin/bundle:23:in `
' Tasks: TOP => install (See full trace by running task with --trace) $ bundle exec gem install /tmp/ruby-c-extension/example_ext/pkg/example_ext-0.1.0.gem ERROR: Error installing /tmp/ruby-c-extension/example_ext/pkg/example_ext-0.1.0.gem: ERROR: Failed to build gem native extension. No such file or directory @ dir_s_mkdir - /home/wsh/.rbenv/versions/3.0.2-dbg/lib/ruby/gems/3.0.0/gems/example_ext-0.1.0/ext/example_ext/.gem.20210813-500914-tvlpga Gem files will remain installed in /home/wsh/.rbenv/versions/3.0.2-dbg/lib/ruby/gems/3.0.0/gems/example_ext-0.1.0 for inspection. Results logged to /home/wsh/.rbenv/versions/3.0.2-dbg/lib/ruby/gems/3.0.0/extensions/x86_64-linux/3.0.0-static/example_ext-0.1.0/gem_make.out ``` So please use Ruby 2.7.4 at first, to avoid such errors. {{< /admonition >}} ## Build CRuby (optional) By this step, we'll be able to debug (step into) `ruby` C source with gdb. It's not necessary but would be greatly help when debugging gems. Here we use [rbenv](https://github.com/rbenv/rbenv) with [ruby-build](https://github.com/rbenv/ruby-build) plugin. Here rbenv is assumed to be installed in `~/.rbenv`. ```sh # --keep keeps source code in ~/.rbenv/sources/2.7.4/ruby-2.7.4/ rbenv install --keep --verbose 2.7.4 rbenv shell 2.7.4 # confirm that ruby is debuggable rbenv which ruby # => /home/wsh/.rbenv/versions/2.7.4/bin/ruby gdb -q ~/.rbenv/versions/2.7.4/bin/ruby ``` In the gdb session, execute the following commands: - `break main` - `run -e 'puts("hello")'` - `info line` - `info source` - `list` - `continue` - `quit` The output should be like: ```log (gdb) break main Breakpoint 1 at 0x1120: file ./main.c, line 38. (gdb) run -e 'puts("hello")' Starting program: /home/wsh/.rbenv/versions/2.7.4/bin/ruby -e 'puts("hello")' [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Breakpoint 1, main (argc=3, argv=0x7fffffffd748) at ./main.c:38 38 { (gdb) info line Line 38 of "./main.c" starts at address 0x555555555120
and ends at 0x555555555149 . (gdb) info source Current source file is ./main.c Compilation directory is /home/wsh/.rbenv/sources/2.7.4/ruby-2.7.4 Located in /home/wsh/.rbenv/sources/2.7.4/ruby-2.7.4/main.c Contains 52 lines. Source language is c. Producer is GNU C99 9.3.0 -mtune=generic -march=x86-64 -ggdb3 -O3 -std=gnu99 -fPIC -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -fasynchronous-unwind-tables -fstack-protector-strong -fstack-clash-protection -fcf-protection. Compiled with DWARF 2 debugging format. Includes preprocessor macro info. (gdb) list 33 #include 34 #endif 35 36 int 37 main(int argc, char **argv) 38 { 39 #ifdef RUBY_DEBUG_ENV 40 ruby_set_debug_option(getenv("RUBY_DEBUG")); 41 #endif 42 #ifdef HAVE_LOCALE_H (gdb) continue Continuing. hello [Inferior 1 (process 360063) exited normally] (gdb) quit ``` Ruby without optimization (`-O0`) is more debuggable: ```sh git clone https://github.com/ruby/ruby.git git checkout v2_7_4 cd ruby/ autoconf # as written in README.md mkdir build; cd build/ # --disable-install-doc saves build time ../configure --prefix=$HOME/.rbenv/versions/2.7.4-dbg --disable-install-doc --enable-debug-env optflags="-O0" make V=1 -j4 make install rbenv shell 2.7.4-dbg rbenv version # 2.7.4-dbg (set by RBENV_VERSION environment variable) ruby --version # ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux] rbenv which ruby # /home/wsh/.rbenv/versions/2.7.4-dbg/bin/ruby gdb -q ~/.rbenv/versions/2.7.4-dbg/bin/ruby -ex 'b main' -ex 'r -e "puts 42"' -ex 'c' -ex 'q' # Breakpoint 1, main (argc=3, argv=0x7fffffffd728) at ../main.c:38 ``` or you can use the tar ball: ```sh wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.4.tar.gz tar xvf ruby-2.7.4.tar.gz cd ruby-2.7.4/ mkdir build/; cd build/ # --disable-install-doc saves build time ../configure --prefix=$HOME/.rbenv/versions/2.7.4-dbg --disable-install-doc --enable-debug-env optflags="-O0" make V=1 -j4 make install rbenv shell 2.7.4-dbg rbenv version # 2.7.4-dbg (set by RBENV_VERSION environment variable) ruby --version # ruby 2.7.4p107 (2021-07-07 revision a21a3b7d23) [x86_64-linux] gdb -q ~/.rbenv/versions/2.7.4-dbg/bin/ruby -ex 'b main' -ex 'r -e "puts 42"' -ex 'c' -ex 'q' # Breakpoint 1, main (argc=3, argv=0x7fffffffd728) at ../main.c:38 ``` - `debugflags="-g"` isn't needed since [`debug=flags=-ggdb3` is default](https://github.com/ruby/ruby/blob/v2_6_1/configure.ac#L655). - [`V=1` shows build commands](https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rules.html). In the following sections, `rbenv shell 2.7.4-dbg` (`~/.rbenv/versions/2.7.4-dbg/`) is supposed to be used. ## Create and build gem ```sh rbenv shell 2.7.4-dbg which bundle # /home/wsh/.rbenv/shims/bundle rbenv which bundle # /home/wsh/.rbenv/versions/2.7.4-dbg/bin/bundle bundle --version # Bundler version 2.1.4 # Enter, Enter, Enter... # Do you want to generate tests with your gem? -> none # Do you want to license your code permissively under the MIT license? -> n # Do you want to include a code of conduct in gems you generate? -> n # Or, of course you can select y, y, y... # These selections will be stored into ~/.bundle/config . bundle gem example_ext --ext cd example_ext/ git commit -m 'bundle gem example_ext --ext' bin/setup # as described in ./README.md ``` `bin/setup` fails with the following error: ```log $ bin/setup bundle install + bundle install You have one or more invalid gemspecs that need to be fixed. The gemspec at /tmp/ruby-c-extension/example_ext/example_ext.gemspec is not valid. Please fix this gemspec. The validation error was 'metadata['homepage_uri'] has invalid link: "TODO: Put your gem's website or public repo URL here."' ``` To fix it, edit `example_ext.gemspec`: ```diff diff --git a/example_ext.gemspec b/example_ext.gemspec index f190b0e..a34275f 100644 --- a/example_ext.gemspec +++ b/example_ext.gemspec @@ -6,16 +6,16 @@ Gem::Specification.new do |spec| spec.authors = ["Wataru Ashihara"] spec.email = ["wataash@wataash.com"] - spec.summary = %q{TODO: Write a short summary, because RubyGems requires one.} - spec.description = %q{TODO: Write a longer description or delete this line.} - spec.homepage = "TODO: Put your gem's website or public repo URL here." + spec.summary = %q{Write a short summary, because RubyGems requires one.} + # spec.description = %q{TODO: Write a longer description or delete this line.} + # spec.homepage = "TODO: Put your gem's website or public repo URL here." spec.required_ruby_version = Gem::Requirement.new(">= 2.3.0") spec.metadata["allowed_push_host"] = "TODO: Set to 'http://mygemserver.com'" - spec.metadata["homepage_uri"] = spec.homepage - spec.metadata["source_code_uri"] = "TODO: Put your gem's public repo URL here." - spec.metadata["changelog_uri"] = "TODO: Put your gem's CHANGELOG.md URL here." + # spec.metadata["homepage_uri"] = spec.homepage + # spec.metadata["source_code_uri"] = "TODO: Put your gem's public repo URL here." + # spec.metadata["changelog_uri"] = "TODO: Put your gem's CHANGELOG.md URL here." # Specify which files should be added to the gem when it is released. # The `git ls-files -z` loads the files in the RubyGem that have been added into git. ``` [View on GitHub](https://github.com/wataash/ruby-example-c-ext/commit/7c06aad6061f742300d5bec29998b8db3308038a) Now the building should succeed: ```sh bin/setup bundle exec rake install # as described in README.md bundle exec ruby -e 'require "example_ext"; p ExampleExt::VERSION' # "0.1.0" ``` ```log $ bin/setup bundle install + bundle install Using rake 12.3.3 Using bundler 2.1.4 Using example_ext 0.1.0 from source at `.` Using rake-compiler 1.1.1 Bundle complete! 3 Gemfile dependencies, 4 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. # Do any other automated setup that you need to do here $ bundle exec rake install mkdir -p tmp/x86_64-linux/example_ext/2.7.4 cd tmp/x86_64-linux/example_ext/2.7.4 /home/wsh/.rbenv/versions/2.7.4-dbg/bin/ruby -I. ../../../../ext/example_ext/extconf.rb creating Makefile cd - cd tmp/x86_64-linux/example_ext/2.7.4 /usr/bin/make compiling ../../../../ext/example_ext/example_ext.c linking shared-object example_ext/example_ext.so cd - mkdir -p tmp/x86_64-linux/stage/lib/example_ext install -c tmp/x86_64-linux/example_ext/2.7.4/example_ext.so lib/example_ext/example_ext.so cp tmp/x86_64-linux/example_ext/2.7.4/example_ext.so tmp/x86_64-linux/stage/lib/example_ext/example_ext.so example_ext 0.1.0 built to pkg/example_ext-0.1.0.gem. example_ext (0.1.0) installed. $ bundle exec ruby -e 'require "example_ext"; p ExampleExt::VERSION' "0.1.0" ``` Let's implement and execute the C extension. Here we define the module method `ExampleExt.hello`. ```diff diff --git a/ext/example_ext/example_ext.c b/ext/example_ext/example_ext.c index c89d90a..f47c72e 100644 --- a/ext/example_ext/example_ext.c +++ b/ext/example_ext/example_ext.c @@ -2,8 +2,18 @@ VALUE rb_mExampleExt; +static VALUE +example_hello(VALUE obj) +{ + printf("hello\n"); + + return Qnil; +} + void Init_example_ext(void) { rb_mExampleExt = rb_define_module("ExampleExt"); + + rb_define_module_function(rb_mExampleExt, "hello", example_hello, 0); } ``` [View on GitHub](https://github.com/wataash/ruby-example-c-ext/commit/356b166c82fffeefd126995592b2520e63225945) ```sh bundle exec rake install bundle exec ruby -e 'require "example_ext"; ExampleExt.hello' # hello ``` ```log $ bundle exec rake install cd tmp/x86_64-linux/example_ext/2.7.4 /usr/bin/make compiling ../../../../ext/example_ext/example_ext.c linking shared-object example_ext/example_ext.so cd - install -c tmp/x86_64-linux/example_ext/2.7.4/example_ext.so lib/example_ext/example_ext.so cp tmp/x86_64-linux/example_ext/2.7.4/example_ext.so tmp/x86_64-linux/stage/lib/example_ext/example_ext.so example_ext 0.1.0 built to pkg/example_ext-0.1.0.gem. example_ext (0.1.0) installed. $ bundle exec ruby -e 'require "example_ext"; ExampleExt.hello' hello ``` [The Definitive Guide to Ruby's C API](http://silverhammermba.github.io/emberb/c/) is a great guide to get started with the C API. ## Debug native extension with gdb Build the native extension without optimization (`-O0`) and with debug symbols (`-ggdb3`). ```sh cd ext/example_ext/ vim extconf.rb # apply diff below ruby extconf.rb make clean && make V=1 # gcc -ggdb3 -O0 make clean cd ../../ bundle exec rake install ``` ```diff diff --git a/ext/example_ext/extconf.rb b/ext/example_ext/extconf.rb index f657c82..7f21db4 100644 --- a/ext/example_ext/extconf.rb +++ b/ext/example_ext/extconf.rb @@ -1,3 +1,13 @@ +require "rbconfig" +if RbConfig::MAKEFILE_CONFIG["CFLAGS"].include?("-g -O2") + fixed_CFLAGS = RbConfig::MAKEFILE_CONFIG["CFLAGS"].sub("-g -O2", "$(cflags)") + puts("Fix CFLAGS: #{RbConfig::MAKEFILE_CONFIG["CFLAGS"]} -> #{fixed_CFLAGS}") + RbConfig::MAKEFILE_CONFIG["CFLAGS"] = fixed_CFLAGS +end + require "mkmf" +CONFIG["optflags"] = "-O0" +CONFIG["debugflags"] = "-ggdb3" + create_makefile("example_ext/example_ext") ``` {{< admonition info "Detailed explanation of this diff" false >}} `CONFIG["optflags"] = "-O0"` and `CONFIG["debugflags"] = "-ggdb3"` changes the Makefile variable `optflags` and `debugflags`, which are used as `CFLAGS`. ```Makefile cflags = $(optflags) $(debugflags) $(warnflags) optflags = -O0 debugflags = -ggdb3 CFLAGS = $(CCDLFLAGS) $(cflags) $(ARCH_FLAG) ``` If you built Ruby with `optflags="-O0"` (and defaultly `debugflags="-ggdb3"`), these substitutions are not needed, since they are set to the ones with which Ruby built, which are defined in `~/.rbenv/versions/2.7.4-dbg/lib/ruby/2.7.0/x86_64-linux/rbconfig.rb`: ```rb CONFIG["debugflags"] = "-ggdb3" CONFIG["optflags"] = "-O0" ``` `CFLAGS` in `rbconfig.rb` is like this: ```rb # 2.7.1: ~/.rbenv/versions/2.7.1/lib/ruby/2.7.0/x86_64-linux/rbconfig.rb CONFIG["CFLAGS"] = "$(cflags) -fPIC" # 2.7.4: ~/.rbenv/versions/2.7.4/lib/ruby/2.7.0/x86_64-linux/rbconfig.rb CONFIG["CFLAGS"] = "-g -O2 -fPIC" # 2.7.4-dbg: ~/.rbenv/versions/2.7.4-dbg/lib/ruby/2.7.0/x86_64-linux/rbconfig.rb CONFIG["CFLAGS"] = "-g -O2" # 3.0.2: ~/.rbenv/versions/3.0.2/lib/ruby/3.0.0/x86_64-linux/rbconfig.rb CONFIG["CFLAGS"] = "$(cflags) -fPIC" ``` And `Makefile`s generated by `ruby extconf.rb` are like this: ```Makefile # 2.7.1 CFLAGS = $(CCDLFLAGS) $(cflags) -fPIC $(ARCH_FLAG) # 2.7.4 CFLAGS = $(CCDLFLAGS) -g -O2 -fPIC $(ARCH_FLAG) # 2.7.4-dbg CFLAGS = $(CCDLFLAGS) -g -O2 $(ARCH_FLAG) # 3.0.2 CFLAGS = $(CCDLFLAGS) $(cflags) -fPIC $(ARCH_FLAG) ``` 2.7.4 seems to have a bug -- `$(cflags)` is ignored, so we can't specify `-ggdb3 -O0` and the extension are always compiled with `-g -O2`. `RbConfig::MAKEFILE_CONFIG["CFLAGS"] = fixed_CFLAGS` fixes this bug. {{< /admonition >}} [View on GitHub](https://github.com/wataash/ruby-example-c-ext/commit/5b63ba60d795eb9fee283bdb2dc9735c43374996) ```log $ ruby extconf.rb Fix CFLAGS: -g -O2 -> $(cflags) creating Makefile $ make clean && make V=1 gcc -I. -I/home/wsh/.rbenv/versions/2.7.4-dbg/include/ruby-2.7.0/x86_64-linux -I/home/wsh/.rbenv/versions/2.7.4-dbg/include/ruby-2.7.0/ruby/backward -I/home/wsh/.rbenv/versions/2.7.4-dbg/include/ruby-2.7.0 -I. -fPIC -ggdb3 -o example_ext.o -c example_ext.c rm -f example_ext.so gcc -shared -o example_ext.so example_ext.o -L. -L/home/wsh/.rbenv/versions/2.7.4-dbg/lib -Wl,-rpath,/home/wsh/.rbenv/versions/2.7.4-dbg/lib -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--compress-debug-sections=zlib -lm -lc $ make clean $ cd ../../ $ bundle exec rake install cd tmp/x86_64-linux/example_ext/2.7.4 /home/wsh/.rbenv/versions/2.7.4-dbg/bin/ruby -I. ../../../../ext/example_ext/extconf.rb Fix CFLAGS: -g -O2 -> $(cflags) creating Makefile cd - cd tmp/x86_64-linux/example_ext/2.7.4 /usr/bin/make linking shared-object example_ext/example_ext.so cd - install -c tmp/x86_64-linux/example_ext/2.7.4/example_ext.so lib/example_ext/example_ext.so cp tmp/x86_64-linux/example_ext/2.7.4/example_ext.so tmp/x86_64-linux/stage/lib/example_ext/example_ext.so example_ext 0.1.0 built to pkg/example_ext-0.1.0.gem. example_ext (0.1.0) installed. ``` Debug it with `gdb`. ```sh # (a) recommended: bundle exec gdb -q -ex 'set breakpoint pending on' -ex 'b example_hello' -ex run --args ruby -e 'require "example_ext"; ExampleExt.hello' # (b) or: env RUBYLIB=./lib gdb -q -ex 'set breakpoint pending on' -ex 'b example_hello' -ex run --args ~/.rbenv/versions/2.7.4-dbg/bin/ruby -e 'require "example_ext"; ExampleExt.hello' # (c) not recommended: gdb -q -ex 'set breakpoint pending on' -ex 'b example_hello' -ex run --args ~/.rbenv/versions/2.7.4-dbg/bin/ruby -e 'require "example_ext"; ExampleExt.hello' ``` {{< admonition info "Detailed explanation about the gdb invocation" false >}} In (a), with `bundle exec` context, `require "example_ext"` (indirectly) loads `/path/to/example_ext/example_ext.so` whose debug info refers to `/path/to/example_ext/ext/example_ext/example_ext.c`. Without `bundle exec` (b), gdb can't load `ruby` which is a shell script, so specify absolute path to Ruby binary. And `RUBY_LIB=./lib` is needed; otherwise (c), `~/.rbenv/versions/2.7.4-dbg/.../example_ext.so` will be loaded which refers to `.rbenv/versions/2.7.4-dbg/.../example_ext.c` [1]. We can [`set substitute-path` gdb command](https://sourceware.org/gdb/onlinedocs/gdb/Source-Path.html#set-substitute_002dpath) to let gdb looks up the original sources. Try commands below for further understanding. ```sh echo $PATH which -a ruby bundle exec echo $PATH bundle exec which -a ruby file ~/.rbenv/shims/ruby file ~/.rbenv/versions/2.7.4-dbg/bin/ruby ruby -e 'p $:' # or $LOAD_PATH bundle exec ruby -e 'p $:' env RUBYLIB=./lib ruby -e 'p $:' ``` [1] The `rake install` task executes the `build` task and `gem install /path/to/example_ext/pkg/example_ext-0.1.0.gem` command. `gem install example_ext-0.1.0.gem` extracts the C sources to `~/.rbenv/versions/2.7.4-dbg/lib/ruby/gems/2.7.0/gems/example_ext-0.1.0/` and compiles them, so debug info for the source path points to `~/.rbenv/versions/2.7.4-dbg/lib/ruby/gems/2.7.0/gems/example_ext-0.1.0/ext/example_ext/example_ext.c`. References: - `bundle exec ruby -rtracer ~/.rbenv/versions/2.7.4-dbg/bin/rake --trace install` - https://github.com/bundler/bundler/blob/v2.0.1/lib/bundler/gem_helper.rb#L46 - https://github.com/bundler/bundler/blob/v2.0.1/lib/bundler/gem_helper.rb#L88 - https://github.com/rubygems/rubygems/blob/v3.0.2/lib/rubygems/installer.rb#L318 - https://github.com/rake-compiler/rake-compiler/blob/v1.0.7/lib/rake/extensiontask.rb#L205 {{< /admonition >}} Now we can debug the C extension! ```log $ bundle exec gdb -q -ex 'set breakpoint pending on' -ex 'b example_hello' -ex run --args ruby -e 'require "example_ext"; ExampleExt.hello' Reading symbols from ruby... Function "example_hello" not defined. Breakpoint 1 (example_hello) pending. Starting program: /home/wsh/.rbenv/versions/2.7.4-dbg/bin/ruby -e require\ \"example_ext\"\;\ ExampleExt.hello [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [Detaching after vfork from child process 802638] Breakpoint 1, example_hello (obj=93824995722496) at ../../../../ext/example_ext/example_ext.c:7 7 { (gdb) list 2 3 VALUE rb_mExampleExt; 4 5 static VALUE 6 example_hello(VALUE obj) 7 { 8 printf("hello\n"); 9 10 return Qnil; 11 } (gdb) next 8 printf("hello\n"); (gdb) next hello 10 return Qnil; (gdb) next 11 } (gdb) next vm_call_cfunc_with_frame (empty_kw_splat=, cd=0x55555607f450, calling=, reg_cfp=0x7ffff71a9fa0, ec=0x5555558a8500) at ../vm_insnhelper.c:2516 2516 CHECK_CFP_CONSISTENCY("vm_call_cfunc"); (gdb) continue Continuing. [Inferior 1 (process 820270) exited normally] (gdb) quit ``` ## References - [rubygems.org - GEMS WITH EXTENSIONS](https://guides.rubygems.org/gems-with-extensions/) - [ruby-doc.org - mkmf](https://ruby-doc.org/stdlib-3.0.2/libdoc/mkmf/rdoc/index.html) - [The Definitive Guide to Ruby's C API](http://silverhammermba.github.io/emberb/c/)