Redis to ValkeyDB migration guide

Redis is an open-source project beloved by developers worldwide, thanks to its performance and ease of use. Its popularity is clear. But recently, changes in Redis's licensing have made people worried about its future and how sustainable it will be. These developments have also highlighted ValkeyDB as a promising fork alternative to Redis that upholds the true spirit of open-source.

The licensing shift in Redis:

On March 20, 2024, Redis announced a change in its licensing model, moving away from the BSD 3-Clause License to a more restrictive dual-license arrangement (RSALv2 and SSPLv1). This change means that companies using Redis to power their services now face stricter rules and may need to either adjust to these new terms or look for other options. This change marks a significant move away from the original, freer ethos of Redis, which was all about providing open and unrestricted access to its software. RSALv2 and SSPLv1 come with restrictions that significantly impact the previous use cases for Redis:

1. RSALv2 (Redis Source Available License version 2): This license allows users to use, modify, and distribute the software, but it restricts commercial use. Specifically, it limits the ability to offer Redis as a part of commercial managed service offerings, which means companies can't use Redis in their cloud services without compliance.
2. SSPLv1 (Server Side Public License version 1): Developed by MongoDB, this license is similar to RSALv2 but goes a step further. It requires that if a company offers Redis as a service, they must also release the entire source code of their service, including all modifications and underlying software that interacts with the database, under the same license.

While it is not articulated directly, it seems like these changes are actually and only targeting the companies that offer competitive services, like  Amazon ElastiCache or Google Cloud MemoryStore.

However, it's important for all users, including those not directly competing in the service space, to understand the broader implications of these changes.

Broader Implications for General Users:

1. Future Feature Limitations: The new licenses may restrict the availability of new features to the broader community under more restrictive terms.
2. Community and Support Changes: The more restrictive nature of these license may potentially reduce contributions from the community.
3. Ecosystem Fragmentation: As service providers respond to these changes, we may see a fragmentation of the Redis ecosystem. Different providers may fork or create alternate versions of Redis (like ValkeyDB), leading to inconsistencies and compatibility challenges for end-users.

Why ValkeyDB?

ValkeyDB is  launched by Linux Foundation and has emerged as a significant fork of Redis. It maintains the principles of open-source software that many developers and organizations value.

ValkeyDB is fully compatible with Redis and also aims to remain that way. It supports the same commands, data types, and features as Redis, making it easy for existing Redis users to switch to ValkeyDB with minimum operational overhead. ValkeyDB not only aims to maintain feature parity with Redis but also introduces enhancements and performance improvements. Here is a video snippet that summarizes these improvements and differences.

Migration from Redis to ValkeyDB

Since, ValkeyDB doesn’t have any major updates in its repository,it’s actually still a Redis. So, this similarity simplifies the migration process, allowing you to treat ValkeyDB as if it were Redis, ensuring ease of transition.

Prepare Your Redis Instance

First, configure your Redis instance to allow replication. This involves modifying your Redis configuration file to include the replicaof line, pointing it towards your ValkeyDB instance.

 replicaof valkeydb_host valkeydb_port

             - If authentication is used, set the masterauth directive to the master's password to allow the replica to authenticate with the master.

2. Backup Redis Data:

- Run redis-cli BGSAVE to save the dataset to disk asynchronously. Since it operates in the background, it doesn’t affect any write or read requests.

Warning: During a BGSAVE, Redis forks a new process, which initially shares the same memory as the parent process due to the copy-on-write method. However, as the database continues to change after the fork, additional memory and disk I/O will be used. This is due to the copy-on-write mechanism, where changes to any data that the child process also references cause the modified data to be copied, thus increasing memory usage. If you’re doing this operation in a busy and operational workload, you need to pick a time when write requests per second are at their lowest.

3- Restart Redis:

- Restart your Redis instance for the changes to take effect.

Configure ValkeyDB as a Replica

In the second step, you will set up ValkeyDB to operate as a replica. This setup requires configuring ValkeyDB to sync data from your Redis master, mirroring all data handling and storage configurations.

2. Configure ValkeyDB Settings: Edit the ValkeyDB configuration file (valkey.conf). Add the following lines to enable replication:

 slaveof   masterauth 

- Start ValkeyDB: Launch ValkeyDB with the updated configuration. You can typically start ValkeyDB using a command like:

 valkey-server /path/to/your/valkey.conf

Initiate and Monitor Data Synchronization

1. Verify Connection: Once ValkeyDB starts, it should automatically begin syncing data from the Redis master. Check the logs of ValkeyDB to ensure it has successfully connected to Redis and started the replication process.
2. Data Synchronization: ValkeyDB will replicate all existing data from Redis as well as any new data changes. This process ensures that ValkeyDB has a complete and up-to-date copy of the Redis dataset.
3. Monitor Replication: Ensure that data remains consistent across both databases throughout this process. The command redis-cli info replication can provide insights into the replication status and health.

Gradual Cutover to ValkeyDB

Switching from Redis to ValkeyDB is an important step that needs careful planning to avoid disrupting your applications. The idea of a gradual cutover is to slowly move your operations from Redis to ValkeyDB, allowing you to monitor and fix any issues that come up along the way. This process involves gradually shifting the read and write operations, checking data consistency, and making sure that both databases work well together until the final switch over.

1. Plan the Cutover:
   
   -Identify Critical Operations: Categorize your Redis operations by their criticality. Start with non-critical operations (e.g., cache lookups) and move towards more critical operations (e.g., transactional data).
   
   - Define Success Metrics: Establish metrics to determine the success of the migration. These could include response times, error rates, and data consistency checks.
   
   - Create a Rollback Plan: Ensure you have a detailed rollback plan in case of any issues during the cutover. This plan should include steps to revert traffic back to Redis seamlessly.
2. Redirect Traffic Gradually:
   
   -Initial Read Operations:
              - Implement Read Proxy: Use a proxy layer (e.g., HAProxy, NGINX) or modify your application to redirect a small fraction (e.g., 5-10%) of read operations to ValkeyDB.
              - Monitor Performance: Track the performance and behavior of ValkeyDB under this initial load. Use monitoring tools to check for response times, error rates, and data consistency.
              - Incremental Increase: Gradually increase the percentage of read traffic to ValkeyDB (e.g., 10-20% increments) while continuously monitoring system performance.
3. Write Operations:
   - Enable Dual Writes: Start by writing data to both Redis and ValkeyDB. This ensures that ValkeyDB receives all new data without interrupting Redis operations. You can Implement this in your application logic. Or if you already’ve, you can use middleware like a queue like Kafka or or a proxy like Twemproxy.
   
   -Monitor Write Consistency
   : Ensure that data written to ValkeyDB matches the data in Redis. Use data validation tools or custom scripts to verify consistency.
   
   - Incremental Write Shift
   : Gradually shift write operations to ValkeyDB. Start with a small portion (e.g., 5-10%) and increase incrementally, similar to the read operations. Monitor for any discrepancies or performance issues.
4. Monitoring and Adjustments:
   
   - Performance Monitoring: Use monitoring tools like Prometheus, Grafana, or built-in tools in ValkeyDB to track metrics such as latency, throughput, and error rates. 
   
   - Data Consistency Checks: Regularly compare data between Redis and ValkeyDB to ensure consistency. Use tools like redis-diff or custom scripts for this purpose. 
   
   - Configuration Adjustments: Based on performance metrics and feedback, adjust configurations in ValkeyDB or your application. This might include tweaking connection settings, optimizing queries, or scaling resources.
   
   - Feedback Loop: Establish a feedback loop with your team to quickly address any issues that arise. This could involve daily stand-up meetings or real-time monitoring dashboards.

Decommission Redis

After successfully migrating to ValkeyDB and ensuring that everything runs smoothly, it's time to say goodbye to Redis. But before you do, there are a few steps to make sure everything is in place and to avoid any issues. Think of it like a final cleanup after a big move. You've packed and moved all your stuff, double-checked that nothing's left behind, and now it's time to close the old door for good.

1. Validate ValkeyDB Operations

• Read-Only Mode for Redis:Transition Redis to read-only and monitor ValkeyDB’s performance.
• Extended Testing:Maintain this setup for a period to ensure stability.

• Shutdown Plan and Final BackupDevelop a detailed plan and perform a final backup.
• Server ShutdownGradually shut down Redis servers, monitoring system impact.

• Continuous Monitoring and AlertingImplement comprehensive monitoring and alerting mechanisms.

By following these steps, you can transition from Redis to ValkeyDB smoothly, ensuring your applications remain robust and performant while adhering to open-source principles.

Considerations for a Smooth Transition

In our migration from Redis to ValkeyDB, we took several crucial steps to ensure a seamless transition and maintain data integrity.

What We Did: We began by setting up ValkeyDB to act as a replica of our Redis instance. This involved configuring Redis to allow replication and ensuring ValkeyDB could communicate with it. We then carefully monitored the synchronization process to make sure all data was accurately replicated from Redis to ValkeyDB. This step was vital to ensure that ValkeyDB had an up-to-date copy of our dataset.

Why We Did It: The primary reason for this replication setup was to minimize downtime and avoid data loss during the migration. By replicating data to ValkeyDB while keeping Redis operational, we ensured that our services remained available and responsive throughout the process. This approach also allowed us to validate that ValkeyDB could handle our data and workload effectively before fully committing to the switch.

How It Is Now: After successfully synchronizing the data, we gradually redirected traffic from Redis to ValkeyDB. Initially, we started with read operations to test the performance and consistency of ValkeyDB under actual usage conditions. Once we were confident in its stability, we began shifting write operations as well. This gradual cutover helped us monitor and address any issues in real-time without impacting our users.

The final step was to decommission Redis. We took this step cautiously, transitioning Redis to read-only mode first to ensure all data changes were captured by ValkeyDB. After an extended period of validation and monitoring, we fully decommissioned Redis, relying solely on ValkeyDB for our database needs.

This migration process was guided by our commitment to maintaining high availability, data integrity, and performance. By carefully planning and executing each step, we were able to achieve a smooth transition with minimal disruption to our services. ValkeyDB now serves as our primary database, providing the open-source flexibility and enhanced performance.

Conclusion

At kloia, a leading software, DevOps, QA, and AI consulting company, we understand the importance of staying ahead with cutting-edge technology while ensuring seamless operations. Migrating from Redis to ValkeyDB not only aligns with open-source principles but also offers enhanced performance and flexibility. By following this comprehensive, zero-downtime migration guide, you can confidently transition to ValkeyDB, leveraging its robust capabilities for your business needs. For further assistance and tailored solutions, feel free to reach out to our team.