Last December, I published version 1 of spider-crab on GitHub.
spider-crab is written in Rust and uses Tokio, clap, reqwest, scraper, and petgraph to crawl across webpages to check for broken links.

The main reason I made spider-crab was to learn Rust and familiarize myself with its ecosystem. If you are actually looking for a higher-grade application to check your website for broken links, I recommend using lychee, which also happens to be written with async Rust.

Around that time I recently learned async Python, so I was excited when I saw that Rust also had async features. I thought that async Rust worked the same as async Python. Async Python has a single thread of execution per event loop, and it cooperatively switches between tasks in the event loop. I did not realize that async Rust may move tasks between threads, so I was extremely frustrated when I saw that many of my variables needed to implement the Send trait.

When I finally understood that tokio performs M:N threading, development became much easier.

Next came the testing, and wow did I have more bugs than I realized! Thankfully Rust makes it super easy to write tests, and provides methods of calculating code coverage. I used mockito to provide HTTP mocks for my unit tests, and created a mini test harness/library to quickly create test cases and reduce duplicated code.

Finally, I split the project into an executable and a library. This took some more research and head scratching, but I eventually figured out how to configure the Cargo.toml to specify both a library and executable target. I have a feeling that the ecosystem of Rust build tools will continue to grow, and I really hope that the Rust standard library absorbs some of these build features. I do not want there to be a separate build tool for Rust, like how C/C++ has CMake/Autotools/Meson.

Overall this was a great first project for learning Rust, and I am thrilled with my Rust experience.