Mirrored from https://github.com/shadow/shadow/discussions/3187:
This is part of a series of periodic updates of development in Shadow. This work is sponsored by the [NSF](https://shadow.github.io/docs/guide/nsf_sponsorship.html). Previous update: [2023-06](https://github.com/shadow/shadow/discussions/3061).
We've merged [74 non-dependabot pull requests](https://github.com/shadow/shadow/pulls?q=is%3Apr+merged%3A2023-06-30..2023-0...) and closed [8 issues](https://github.com/shadow/shadow/issues?q=closed%3A2023-06-30..2023-09-27+is...) since our previous update.
# Latest improvements
## Spawning processes
Shadow now supports running programs that spawn additional processes. Technically, this means it now supports `fork`, `vfork`, `execve`, and other related syscalls. Some uses for this:
* The primary way for `tor` to work with [pluggable transports](https://blog.torproject.org/tor-heart-bridges-and-pluggable-transports/) is for the `tor` process to dynamically spawn the pluggable transport process. This is currently the *only* way that pluggable transports are supported in `arti`. This now works in `shadow`.
* Orchestration of multiple processes on a single host in `shadow` can now be done using e.g. `sh` or `python` scripts, instead of having to specify every process directly in `shadow`'s configuration file.
* `shadow` can now run other software that spawns worker or helper processes.
Another exciting application for this in `shadow` *development* is that we can now more easily run third party test suites, which typically spawn multiple test processes. For example, running `tor`'s own self-tests helps validate that `shadow` is correctly emulating the platform functionality that `tor` uses; currently [more than 99% of the tests pass](https://github.com/shadow/shadow/issues/3168)!
## New TCP stack
Shadow's experimental new TCP stack is merged, and can be enabled with the experimental command-line flag `--use-new-tcp`. This new implementation is in Rust instead of C, and is developed to be very testable (e.g. in its own crate decoupled from the rest of Shadow, and with pluggable system dependencies). While some functionality is still being finished, we expect it will be much easier to maintain and to validate its correctness than the previous C implementation. Improved support for pcap file output also makes it easier to review Shadow's simulated network traffic.
## Shim stability
We've occasionally run into problems due to `shadow`'s preloaded shim calling into `libc`. This is unsafe because some initialization can run before `libc` itself is fully initialized, and much of the code runs in a seccomp signal handler, running afoul of [async-signal-safety](https://man7.org/linux/man-pages/man7/signal-safety.7.html).
Luckily, Rust has a rich ecosystem of code that doesn't depend on libc (`no_std` code). We've made substantial progress in migrating the shim's C code to `no_std` Rust code.
## Socket API improvements
Our UDP socket implementation has been migrated from C to Rust, and along the way we've made many improvements to the socket API for UDP, TCP, and Unix sockets. We've added support for the following:
* `sendmsg`, `recvmsg`, and `shutdown` syscall support for UDP sockets * `MSG_TRUNC` support for UDP and Unix sockets * `MSG_PEEK` support for UDP sockets * `SO_DOMAIN`, `SO_PROTOCOL`, and `SO_ACCEPTCONN` socket options for TCP and UDP sockets * `SIOCGSTAMP` ioctl support for TCP and UDP sockets
We've also made various fixes and improvements to existing socket functionality so that Shadow more accurately follows the behaviour of Linux.
## C to Rust migration
We continue to progress our migration from C to Rust. In addition to Rust progress in the shim and in the new TCP stack, other notable migrations since the last update include the shared memory allocator, epoll and timerfd descriptors and syscalls, UDP sockets, and several time-related syscalls. Our current status is 74% Rust code and 21.4% C code (much of this being C tests) according to Github statistics.
# Release status
We expect to release Shadow 3.1.0 some time in the next few weeks.
# Project status - NSF grant is wrapping up
The last three years of development on Shadow have been sponsored by an [NSF](https://shadow.github.io/docs/guide/nsf_sponsorship.html) grant. That grant ends at the end of this month (September 2023). This isn't the end of development on Shadow — Rob Jansen (@robgjansen) at the U.S. Naval Research Laboratory continues to head the project. Over the course of this project, The Tor Project has incorporated Shadow into its own research and testing workflows, and is likely to continue contributing as well. We look forward to continue improving and maintaining Shadow, but at a slower pace of development since we no longer expect contributions from programmers singularly dedicated to Shadow development. Jim Newsome (@sporksmith) is continuing to be a Tor Project employee but shifting focus to other Tor projects; Steven Engler (@stevenengler) is open to opportunities.
Happy simulating! The Shadow team