script without GPS/capcode lookup

Hello @cyberjunky Ron
I'm still using your script for years now, it's extremely stable thanks for that. I process the data via node-red which creates push messages and TTS calls. I like to improve the speed of processing messages. Would you be able to clean out code and only provide messages to MQTT/ HA WebHook without the db/plaatsnaam/geo lookups to improve speed? Its quite a time offer I ask you, but this holiday season could be a fit ;-) I only need the P2000 text, and capcodes.
I have tried it via AI to do it myself but that ends in receiving 0 messages ;(
https://github.com/cyberjunky/RTL-SDR-P2000Receiver-HA

In the config you can set the opencage/gps lookup on false.
And I don't know on what kind device you run this, but I have max 1-2 seconds delay before the message is accepted by Home Assistant or MQTT.
How much speed do you want?
P2000 messages are already delayed because the transmit interval is not realtime.

Good morning, I have a redundant receiver environment. 1 feed from pagermon, and 1 feed via cyberjunky. The feed from pagermon (webhook) is very fast. That made me realize that I actually wanted a bit more speed via the python script. My wish is: near realtime.
That seems to have worked via claude.ai the tool has created a script that now forwards all decoded messages in real time. After that I asked to add performance optimizations such as multithread and a optimized MQTT handler etc. The tool provided the hint to add performance benchmarks in the code so that we can really see if we are making steps in speed.
The delay between pagermon and the original cyberjunky script is 2seconds and 2 milliseconds. That seems to be 0 delay right now if I compare the log timestamps.
Older script already enhanced by Claude.ai
Performance Statistics:
2024-12-29 00:00:24 - INFO -
mqtt_post:
2024-12-29 00:00:24 - INFO -   Average time: 0.61ms
2024-12-29 00:00:24 - INFO -   Min time: 0.27ms
2024-12-29 00:00:24 - INFO -   Max time: 1.89ms
2024-12-29 00:00:24 - INFO -   Message count: 13
2024-12-29 00:00:24 - INFO -
message_processing:
2024-12-29 00:00:24 - INFO -   Average time: 0.96ms
2024-12-29 00:00:24 - INFO -   Min time: 0.47ms
2024-12-29 00:00:24 - INFO -   Max time: 2.09ms
2024-12-29 00:00:24 - INFO -   Message count: 13
2024-12-29 00:00:24 - INFO - Receiver stopped

Current running version
mqtt_post:
2024-12-29 00:12:33 - INFO -   Average time: 0.30ms
2024-12-29 00:12:33 - INFO -   Min time: 0.19ms
2024-12-29 00:12:33 - INFO -   Max time: 0.88ms
2024-12-29 00:12:33 - INFO -   Message count: 24
2024-12-29 00:12:33 - INFO -
message_processing:
2024-12-29 00:12:33 - INFO -   Average time: 0.60ms
2024-12-29 00:12:33 - INFO -   Min time: 0.40ms
2024-12-29 00:12:33 - INFO -   Max time: 1.31ms
2024-12-29 00:12:33 - INFO -   Message count: 24
2024-12-29 00:12:33 - INFO - Receiver stopped

#!/usr/bin/env python3
import requests.adapters
import configparser
import json
import logging
import os
import re
import subprocess
import sys
import threading
import time
import queue
from collections import deque, defaultdict
from datetime import datetime
from threading import Event
from statistics import mean, median, stdev
from typing import Dict, List, Any

import paho.mqtt.client as mqtt
import requests

VERSION = "0.5.0"
CFGFILE = "config.ini"

# Simple pattern matching pipe-separated fields
FLEX_PATTERN = re.compile(
    r"^FLEX\|([^|]*)\|([^|]*)\|([^|]*)\|([^|]*)\|ALN\|([^|]*)"
)

class PerformanceMetrics:
    """Enhanced performance monitoring with detailed statistics."""

    def __init__(self):
        self.metrics: Dict[str, List[float]] = defaultdict(list)
        self.start_times: Dict[str, float] = {}
        self.message_latencies: List[float] = []
        self.processing_times: List[float] = []
        self.mqtt_delivery_times: List[float] = []

    def start(self, operation: str) -> None:
        """Start timing an operation."""
        self.start_times[operation] = time.perf_counter()

    def stop(self, operation: str) -> None:
        """Stop timing an operation and record duration."""
        if operation in self.start_times:
            duration = time.perf_counter() - self.start_times[operation]
            self.metrics[operation].append(duration)
            del self.start_times[operation]

    def record_latency(self, start_time: float) -> None:
        """Record message delivery latency."""
        latency = time.perf_counter() - start_time
        self.message_latencies.append(latency)

    def record_processing_time(self, duration: float) -> None:
        """Record message processing time."""
        self.processing_times.append(duration)

    def record_mqtt_delivery(self, duration: float) -> None:
        """Record MQTT delivery time."""
        self.mqtt_delivery_times.append(duration)

    def get_stats(self) -> Dict[str, Any]:
        """Return comprehensive performance statistics."""
        stats = {}

        # General operations stats
        for op, times in self.metrics.items():
            if times:
                stats[op] = {
                    "avg_ms": mean(times) * 1000,
                    "median_ms": median(times) * 1000,
                    "min_ms": min(times) * 1000,
                    "max_ms": max(times) * 1000,
                    "std_dev_ms": stdev(times) * 1000 if len(times) > 1 else 0,
                    "count": len(times),
                }

        # Message delivery latency stats
        if self.message_latencies:
            stats["message_latency"] = {
                "avg_ms": mean(self.message_latencies) * 1000,
                "median_ms": median(self.message_latencies) * 1000,
                "min_ms": min(self.message_latencies) * 1000,
                "max_ms": max(self.message_latencies) * 1000,
                "std_dev_ms": (
                    stdev(self.message_latencies) * 1000
                    if len(self.message_latencies) > 1
                    else 0
                ),
                "count": len(self.message_latencies),
            }

        # Processing time stats
        if self.processing_times:
            stats["processing_time"] = {
                "avg_ms": mean(self.processing_times) * 1000,
                "median_ms": median(self.processing_times) * 1000,
                "min_ms": min(self.processing_times) * 1000,
                "max_ms": max(self.processing_times) * 1000,
                "std_dev_ms": (
                    stdev(self.processing_times) * 1000
                    if len(self.processing_times) > 1
                    else 0
                ),
                "count": len(self.processing_times),
            }

        # MQTT delivery stats
        if self.mqtt_delivery_times:
            stats["mqtt_delivery"] = {
                "avg_ms": mean(self.mqtt_delivery_times) * 1000,
                "median_ms": median(self.mqtt_delivery_times) * 1000,
                "min_ms": min(self.mqtt_delivery_times) * 1000,
                "max_ms": max(self.mqtt_delivery_times) * 1000,
                "std_dev_ms": (
                    stdev(self.mqtt_delivery_times) * 1000
                    if len(self.mqtt_delivery_times) > 1
                    else 0
                ),
                "count": len(self.mqtt_delivery_times),
            }

        return stats

class ThreadManager:
    """Manages multiple worker threads and their lifecycle."""

    def __init__(self):
        self.threads = {}
        self.running = threading.Event()
        self.running.set()
        self.performance_metrics = PerformanceMetrics()

    def start_thread(self, name: str, target, daemon: bool = True) -> bool:
        """Start a new named thread."""
        if name in self.threads:
            logging.warning(f"Thread {name} already exists")
            return False

        thread = threading.Thread(
            target=self._thread_wrapper, args=(name, target), daemon=daemon
        )
        self.threads[name] = thread
        thread.start()
        logging.info(f"Started thread: {name}")
        return True

    def _thread_wrapper(self, name: str, target) -> None:
        """Wrapper to handle thread exceptions and cleanup."""
        try:
            logging.info(f"Thread {name} starting")
            self.performance_metrics.start(f"thread_{name}")
            target()
        except Exception as e:
            logging.error(f"Error in thread {name}: {str(e)}")
            logging.exception("Thread error details:")
        finally:
            self.performance_metrics.stop(f"thread_{name}")
            logging.info(f"Thread {name} stopped")
            if name in self.threads:
                del self.threads[name]

    def stop_all(self, timeout: float = 5) -> None:
        """Stop all threads gracefully."""
        self.running.clear()
        start_time = time.time()

        # Wait for threads to finish
        active_threads = list(self.threads.values())
        for thread in active_threads:
            remaining_time = max(0, timeout - (time.time() - start_time))
            if remaining_time > 0 and thread.is_alive():
                thread.join(timeout=remaining_time)

        # Check for any threads that didn't stop
        still_running = [t.name for t in active_threads if t.is_alive()]
        if still_running:
            logging.warning(f"Threads still running: {still_running}")

        self.threads.clear()

class MessageQueue:
    """Thread-safe message queue with priority support and performance monitoring."""

    def __init__(self, maxlen: int = 100):
        self.queue = deque(maxlen=maxlen)
        self.lock = threading.Lock()
        self.not_empty = threading.Condition(self.lock)
        self.not_full = threading.Condition(self.lock)
        self.metrics = PerformanceMetrics()

    def put(self, item, block: bool = True, timeout: float = None) -> bool:
        """Add an item to the queue with performance tracking."""
        self.metrics.start("queue_put")
        try:
            with self.lock:
                if block:
                    while len(self.queue) >= self.queue.maxlen:
                        if not self.not_full.wait(timeout):
                            raise queue.Full
                elif len(self.queue) >= self.queue.maxlen:
                    raise queue.Full

                self.queue.appendleft(item)
                self.not_empty.notify()
                return True
        finally:
            self.metrics.stop("queue_put")

    def get(self, block: bool = True, timeout: float = None):
        """Remove and return an item from the queue with performance tracking."""
        self.metrics.start("queue_get")
        try:
            with self.lock:
                if block:
                    while not self.queue:
                        if not self.not_empty.wait(timeout):
                            raise queue.Empty
                elif not self.queue:
                    raise queue.Empty

                item = self.queue.pop()
                self.not_full.notify()
                return item
        finally:
            self.metrics.stop("queue_get")

    def __len__(self) -> int:
        """Return the current size of the queue."""
        with self.lock:
            return len(self.queue)

class MQTTHandler:
    """Handles MQTT connection and message delivery with improved reliability and performance."""

    def __init__(self, config):
        self.config = config
        self.client = None
        self.connected = False
        self.reconnect_delay = 1  # Start with 1 second delay
        self.max_reconnect_delay = (
            60  # Maximum delay between reconnection attempts
        )
        self.pending_messages = deque(
            maxlen=1000
        )  # Store messages during disconnections
        self.metrics = PerformanceMetrics()
        self.setup_client()

    def setup_client(self):
        """Initialize MQTT client with all necessary callbacks."""
        try:
            self.client = mqtt.Client()

            # Set up authentication
            if self.config["user"] and self.config["password"]:
                self.client.username_pw_set(
                    self.config["user"], self.config["password"]
                )

            # Configure callbacks
            self.client.on_connect = self.on_connect
            self.client.on_disconnect = self.on_disconnect
            self.client.on_publish = self.on_publish

            # Configure QoS and other settings
            self.client.max_queued_messages_set(100)
            self.client.max_inflight_messages_set(20)

            # Enable built-in reconnect functionality
            self.client.reconnect_delay_set(min_delay=1, max_delay=60)

            self.metrics.start("mqtt_connect")
            self.client.connect(
                self.config["server"], self.config["port"], keepalive=60
            )
            self.client.loop_start()
            logging.info("MQTT client setup completed successfully")
        except Exception as e:
            logging.error(f"MQTT connection failed: {e}")
            self.connected = False
        finally:
            self.metrics.stop("mqtt_connect")

    def on_connect(self, client, userdata, flags, rc):
        """Handle connection events and retry sending pending messages."""
        if rc == 0:
            self.connected = True
            self.reconnect_delay = 1
            logging.info("Connected to MQTT broker")

            # Try to send any pending messages
            while self.pending_messages and self.connected:
                topic, payload, qos, start_time = (
                    self.pending_messages.popleft()
                )
                self.publish(topic, payload, qos, start_time)
        else:
            self.connected = False
            logging.error(f"MQTT connection failed with code {rc}")

    def on_disconnect(self, client, userdata, rc):
        """Handle disconnection events with exponential backoff."""
        self.connected = False
        if rc != 0:
            logging.warning(f"MQTT disconnected unexpectedly with code {rc}")
            self.reconnect_delay = min(
                self.reconnect_delay * 2, self.max_reconnect_delay
            )
        else:
            logging.info("MQTT disconnected normally")

    def on_publish(self, client, userdata, mid):
        """Confirm successful message publication."""
        self.metrics.stop(f"mqtt_publish_{mid}")
        logging.debug(f"Message {mid} published successfully")

    def publish(self, topic, payload, qos=1, start_time=None):
        """Publish message with QoS and performance tracking."""
        metric_name = f"mqtt_publish_{self.client._client_id}"  # Define metric name outside try block
        self.metrics.start(metric_name)
        try:
            if not start_time:
                start_time = time.perf_counter()

            if not self.connected:
                self.pending_messages.append((topic, payload, qos, start_time))
                logging.warning("MQTT not connected, message queued")
                return False

            info = self.client.publish(topic, payload, qos=qos)

            if info.rc != mqtt.MQTT_ERR_SUCCESS:
                self.pending_messages.append((topic, payload, qos, start_time))
                logging.error(f"Failed to publish message: {info.rc}")
                return False

            self.metrics.record_mqtt_delivery(time.perf_counter() - start_time)
            return True
        except Exception as e:
            logging.error(f"MQTT Error: {e}")
            self.pending_messages.append((topic, payload, qos, start_time))
            return False
        finally:
            self.metrics.stop(metric_name)  # Ensure metric is always stopped

    def stop(self):
        """Clean shutdown of MQTT connection."""
        if self.client:
            try:
                self.client.loop_stop()
                self.client.disconnect()
            except Exception as e:
                logging.error(f"Error during MQTT shutdown: {e}")

class MessageItem:
    """Message container with performance tracking."""

    __slots__ = (
        "timestamp",
        "message_raw",
        "body",
        "capcodes",
        "receivers",
        "is_posted",
        "creation_time",
    )

    def __init__(self, message_raw, timestamp, body, capcodes):
        self.timestamp = timestamp
        self.message_raw = message_raw
        self.body = body
        self.capcodes = capcodes
        self.receivers = " ".join(capcodes)
        self.is_posted = False
        self.creation_time = time.perf_counter()

def load_config():
    """Load or create config file."""
    config = configparser.ConfigParser()
    if config.read(CFGFILE):
        return config

    config["main"] = {"debug": "true", "benchmark": "true"}
    config["rtl-sdr"] = {
        "cmd": "rtl_fm -f 169.65M -M fm -s 22050 | multimon-ng -a FLEX -t raw -"
    }
    config["home-assistant"] = {
        "enabled": "true",
        "baseurl": "http://homeassistant.local:8123",
        "token": "YOUR_TOKEN_HERE",
    }
    config["mqtt"] = {
        "enabled": "true",
        "mqtt_server": "<IP_ADDRESS>",
        "mqtt_port": "1883",
        "mqtt_user": "mqttuser",
        "mqtt_password": "password",
        "mqtt_topic": "p2000",
        "mqtt_qos": "1",
    }

    with open(CFGFILE, "w") as f:
        config.write(f)
        logging.info(f"Created new config file: {CFGFILE}")
    return config

class P2000Receiver:
    def __init__(self):
        print(f"Initializing P2000 Receiver v{VERSION}...")
        self.thread_manager = ThreadManager()
        self.messages = MessageQueue(maxlen=100)
        self.config = load_config()
        self.mqtt_handler = None
        self.metrics = PerformanceMetrics()

        # Configure logging
        self.debug = self.config.getboolean("main", "debug")
        self.benchmark = self.config.getboolean(
            "main", "benchmark", fallback=True
        )

        log_level = logging.DEBUG if self.debug else logging.INFO
        logging.basicConfig(
            level=log_level,
            format="%(asctime)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S",
        )

        logging.info("Loading configuration...")

        # Load config settings
        self.rtlfm_cmd = self.config.get("rtl-sdr", "cmd")
        logging.info(f"RTL-SDR command: {self.rtlfm_cmd}")

        # Initialize Home Assistant
        self.use_hass = self.config.getboolean("home-assistant", "enabled")
        if self.use_hass:
            self.baseurl = self.config.get("home-assistant", "baseurl")
            self.token = self.config.get("home-assistant", "token")
            self.setup_home_assistant()

        # Initialize MQTT
        self.use_mqtt = self.config.getboolean("mqtt", "enabled")
        if self.use_mqtt:
            self.setup_mqtt()

        logging.info("Initialization complete.")

    def setup_home_assistant(self):
        """Initialize Home Assistant connection."""
        try:
            logging.info("Setting up Home Assistant connection...")
            self.http_session = requests.Session()
            adapter = requests.adapters.HTTPAdapter(
                pool_connections=1,
                pool_maxsize=1,
                max_retries=3,
                pool_block=False,
            )
            self.http_session.mount("http://", adapter)
            self.http_session.mount("https://", adapter)
            self.http_session.headers.update(
                {
                    "Authorization": f"Bearer {self.token}",
                    "content-type": "application/json",
                    "Connection": "keep-alive",
                }
            )
            logging.info("Home Assistant connection setup complete")
        except Exception as e:
            logging.error(f"Error setting up Home Assistant: {e}")
            self.use_hass = False

    def setup_mqtt(self):
        """Initialize MQTT connection."""
        try:
            logging.info("Setting up MQTT connection...")
            self.mqtt_config = {
                "server": self.config.get("mqtt", "mqtt_server"),
                "port": self.config.getint("mqtt", "mqtt_port"),
                "user": self.config.get("mqtt", "mqtt_user"),
                "password": self.config.get("mqtt", "mqtt_password"),
                "topic": self.config.get("mqtt", "mqtt_topic"),
                "qos": self.config.getint("mqtt", "mqtt_qos", fallback=1),
            }
            self.mqtt_handler = MQTTHandler(self.mqtt_config)
            logging.info("MQTT connection setup complete")
        except Exception as e:
            logging.error(f"Error setting up MQTT: {e}")
            self.use_mqtt = False

    def post_message(self, msg):
        """Post message to configured endpoints with performance tracking."""
        self.metrics.start("message_processing")
        processing_start = time.perf_counter()

        try:
            data = {
                "state": msg.body,
                "attributes": {
                    "time_received": msg.timestamp,
                    "receivers": msg.receivers,
                    "capcodes": msg.capcodes,
                    "raw_message": msg.message_raw,
                },
            }

            json_str = json.dumps(data)
            logging.debug(f"Prepared message data: {json_str}")

            # Post to Home Assistant
            if self.use_hass:
                self.metrics.start("ha_post")
                try:
                    response = self.http_session.post(
                        f"{self.baseurl}/api/states/sensor.p2000",
                        data=json_str,
                        timeout=5,
                    )
                    response.raise_for_status()
                except Exception as e:
                    logging.error(f"HA Error: {e}")
                finally:
                    self.metrics.stop("ha_post")

            # Post to MQTT
            if self.use_mqtt and self.mqtt_handler:
                try:
                    topic = f"{self.mqtt_config['topic']}/sensor/p2000"
                    self.mqtt_handler.publish(
                        topic,
                        json_str,
                        qos=self.mqtt_config["qos"],
                        start_time=msg.creation_time,
                    )
                except Exception as e:
                    logging.error(f"MQTT Error: {e}")

            # Record end-to-end processing time
            self.metrics.record_processing_time(
                time.perf_counter() - processing_start
            )

        except Exception as e:
            logging.error(f"Error in post_message: {e}")
        finally:
            self.metrics.stop("message_processing")

    def process_messages(self):
        """Process messages from queue."""
        logging.info("Message processing thread started")
        while self.thread_manager.running.is_set():
            try:
                msg = self.messages.get(block=True, timeout=1.0)
                if not msg.is_posted:
                    self.metrics.record_latency(msg.creation_time)
                    self.post_message(msg)
                    msg.is_posted = True
            except queue.Empty:
                continue
            except Exception as e:
                logging.error(f"Error processing message: {e}")
                time.sleep(1)

    def read_rtl_sdr(self):
        """Read and process RTL-SDR data."""
        logging.info("Starting RTL-SDR process...")
        process = None
        try:
            process = subprocess.Popen(
                self.rtlfm_cmd,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
                shell=True,
                bufsize=1,
                universal_newlines=True,
            )

            logging.info("RTL-SDR process started")
            print("Listening for P2000 messages...")

            while self.thread_manager.running.is_set():
                if process.poll() is not None:
                    raise Exception("RTL-SDR process terminated unexpectedly")

                line = process.stdout.readline()
                if not line:
                    stderr_line = process.stderr.readline()
                    if stderr_line:
                        logging.error(f"RTL-SDR Error: {stderr_line.strip()}")
                    continue

                if "FLEX" in line and "ALN" in line:
                    self.metrics.start("message_parse")
                    try:
                        match = FLEX_PATTERN.search(line)
                        if match:
                            self.handle_message(line, match)
                    finally:
                        self.metrics.stop("message_parse")

        except Exception as e:
            logging.error(f"Error in read_rtl_sdr: {e}")
            print(f"Error: {str(e)}")
        finally:
            if process:
                process.kill()
                process.wait()

    def handle_message(self, line, match):
        """Process and queue a new message."""
        self.metrics.start("message_parse")
        try:
            timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
            capcodes = match.group(4).strip().split()
            message = match.group(5).strip()

            if message and capcodes:
                msg = MessageItem(
                    message_raw=line.strip(),
                    timestamp=timestamp,
                    body=message,
                    capcodes=capcodes,
                )
                self.messages.put(msg, block=False)
                print(f"[{timestamp}] New message: {message}")
        except Exception as e:
            logging.error(f"Error handling message: {e}")
        finally:
            self.metrics.stop(
                "message_parse"
            )  # Ensure metric is always stopped

    def print_performance_stats(self):
        """Print detailed performance statistics."""
        if not self.benchmark:
            return

        stats = {
            "receiver": self.metrics.get_stats(),
            "queue": self.messages.metrics.get_stats(),
            "mqtt": (
                self.mqtt_handler.metrics.get_stats()
                if self.mqtt_handler
                else {}
            ),
            "threads": self.thread_manager.performance_metrics.get_stats(),
        }

        logging.info("\nPerformance Statistics:")
        for component, metrics in stats.items():
            if metrics:
                logging.info(f"\n{component.upper()} METRICS:")
                for op, values in metrics.items():
                    logging.info(f"\n{op}:")
                    # This should be indented to be inside the op loop
                    for metric, value in values.items():
                        if isinstance(value, float):
                            logging.info(f"  {metric}: {value:.2f}ms")
                        else:
                            logging.info(f"  {metric}: {value}")

    def run(self):
        """Start the receiver with performance monitoring."""
        logging.info(f"P2000 Receiver v{VERSION} starting...")
        if self.benchmark:
            logging.info("Performance monitoring enabled")
            self.metrics.start("total_runtime")

        try:
            # Start message processing thread
            self.thread_manager.start_thread("processor", self.process_messages)

            # Start RTL-SDR reading in separate thread
            self.thread_manager.start_thread("rtl-sdr", self.read_rtl_sdr)

            # Wait for shutdown signal
            try:
                while self.thread_manager.running.is_set():
                    time.sleep(1)
            except KeyboardInterrupt:
                logging.info("Shutdown signal received")

        finally:
            # Cleanup and stop all threads
            self.thread_manager.stop_all(timeout=5)
            if self.mqtt_handler:
                self.mqtt_handler.stop()
            if self.benchmark:
                self.metrics.stop("total_runtime")
                self.print_performance_stats()
            logging.info("Receiver stopped")

def main():
    """Main entry point with error handling."""
    try:
        # Set up signal handlers
        def signal_handler(signum, frame):
            logging.info(f"Received signal {signum}")
            if receiver:
                receiver.thread_manager.running.clear()

        import signal

        signal.signal(signal.SIGINT, signal_handler)
        signal.signal(signal.SIGTERM, signal_handler)

        # Create and run receiver
        receiver = P2000Receiver()
        receiver.run()

    except KeyboardInterrupt:
        print("\nShutting down...")
    except Exception as e:
        print(f"Fatal error: {str(e)}")
        logging.exception("Fatal error occurred")
        sys.exit(1)

if __name__ == "__main__":
    main()

This is a stable version of the new code, where I have the feeling it holds up during the new year eave load of the P2000 network
#!/usr/bin/env python3
import requests.adapters
import configparser
import json
import logging
import os
import re
import subprocess
import sys
import threading
import time
import queue
from collections import deque
from datetime import datetime
from threading import Event

import paho.mqtt.client as mqtt
import requests

VERSION = "0.7.1"
CFGFILE = "config.ini"

# Simple pattern matching pipe-separated fields
FLEX_PATTERN = re.compile(
    r"^FLEX\|([^|]*)\|([^|]*)\|([^|]*)\|([^|]*)\|ALN\|([^|]*)"
)

class ThreadManager:
    """Manages multiple worker threads and their lifecycle."""

    def __init__(self):
        self.threads = {}
        self.running = threading.Event()
        self.running.set()

    def start_thread(self, name: str, target, daemon: bool = True) -> bool:
        """Start a new named thread."""
        if name in self.threads:
            logging.warning(f"Thread {name} already exists")
            return False

        thread = threading.Thread(
            target=self._thread_wrapper, args=(name, target), daemon=daemon
        )
        self.threads[name] = thread
        thread.start()
        logging.info(f"Started thread: {name}")
        return True

    def _thread_wrapper(self, name: str, target) -> None:
        """Wrapper to handle thread exceptions and cleanup."""
        try:
            logging.info(f"Thread {name} starting")
            target()
        except Exception as e:
            logging.error(f"Error in thread {name}: {str(e)}")
            logging.exception("Thread error details:")
        finally:
            logging.info(f"Thread {name} stopped")
            if name in self.threads:
                del self.threads[name]

    def stop_all(self, timeout: float = 5) -> None:
        """Stop all threads gracefully."""
        self.running.clear()
        start_time = time.time()

        # Wait for threads to finish
        active_threads = list(self.threads.values())
        for thread in active_threads:
            remaining_time = max(0, timeout - (time.time() - start_time))
            if remaining_time > 0 and thread.is_alive():
                thread.join(timeout=remaining_time)

        # Check for any threads that didn't stop
        still_running = [t.name for t in active_threads if t.is_alive()]
        if still_running:
            logging.warning(f"Threads still running: {still_running}")

        self.threads.clear()

class MessageQueue:
    """Thread-safe message queue with priority support and optimized startup handling."""

    def __init__(self, maxlen: int = 100):
        self.queue = deque(maxlen=maxlen)
        self.lock = threading.Lock()
        self.not_empty = threading.Condition(self.lock)
        self.not_full = threading.Condition(self.lock)
        self.startup_mode = True
        self.startup_timeout = 0.05
        self._startup_time = time.time()

    def put(self, item, block: bool = True, timeout: float = None) -> bool:
        """Add an item to the queue with optimized startup handling."""
        with self.lock:
            if block:
                while len(self.queue) >= self.queue.maxlen:
                    if not self.not_full.wait(timeout):
                        raise queue.Full
            elif len(self.queue) >= self.queue.maxlen:
                raise queue.Full

            self.queue.appendleft(item)
            self.not_empty.notify()
            return True

    def get(self, block: bool = True, timeout: float = None):
        """Remove and return an item from the queue with startup optimization."""
        with self.lock:
            if block:
                current_timeout = self.startup_timeout if self.startup_mode else timeout
                while not self.queue:
                    if not self.not_empty.wait(current_timeout):
                        if self.startup_mode and time.time() - self._startup_time > 30:
                            self.startup_mode = False
                        raise queue.Empty
            elif not self.queue:
                raise queue.Empty

            item = self.queue.pop()
            self.not_full.notify()
            return item

    def exit_startup_mode(self):
        """Explicitly exit startup mode."""
        with self.lock:
            self.startup_mode = False

    def __len__(self) -> int:
        """Return the current size of the queue."""
        with self.lock:
            return len(self.queue)

class MQTTHandler:
    """Handles MQTT connection and message delivery with improved reliability and logging."""

    def __init__(self, config):
        self.config = config
        self.client = None
        self.connected = False
        self.reconnect_delay = 1
        self.max_reconnect_delay = 60
        self.connection_attempts = 0
        self.last_connection_time = None
        self.pending_messages = deque(maxlen=1000)
        self.setup_client()

    def setup_client(self):
        """Initialize MQTT client with all necessary callbacks."""
        try:
            self.client = mqtt.Client(client_id=f"p2000_receiver_{int(time.time())}")

            if self.config["user"] and self.config["password"]:
                self.client.username_pw_set(
                    self.config["user"], self.config["password"]
                )
                logging.info(f"MQTT authentication configured for user: {self.config['user']}")
            else:
                logging.warning("MQTT running without authentication")

            self.client.on_connect = self.on_connect
            self.client.on_disconnect = self.on_disconnect
            self.client.on_publish = self.on_publish
            self.client.on_socket_open = self.on_socket_open
            self.client.on_socket_close = self.on_socket_close
            self.client.on_socket_error = self.on_socket_error

            self.client.max_queued_messages_set(100)
            self.client.max_inflight_messages_set(20)
            self.client.reconnect_delay_set(min_delay=1, max_delay=60)

            logging.info(f"Attempting MQTT connection to {self.config['server']}:{self.config['port']}")
            self.client.connect(
                self.config["server"],
                self.config["port"],
                keepalive=60
            )
            self.client.loop_start()
            logging.info("MQTT client setup completed successfully")

        except ConnectionRefusedError:
            logging.error(f"MQTT connection refused by {self.config['server']}:{self.config['port']}")
            self.connected = False
        except TimeoutError:
            logging.error(f"MQTT connection timeout to {self.config['server']}:{self.config['port']}")
            self.connected = False
        except Exception as e:
            logging.error(f"MQTT connection failed: {str(e)}")
            logging.exception("Full MQTT connection error details:")
            self.connected = False

    def on_socket_open(self, client, userdata, sock):
        """Log when socket is opened."""
        logging.info(f"MQTT socket opened to {self.config['server']}:{self.config['port']}")

    def on_socket_close(self, client, userdata, sock):
        """Log when socket is closed."""
        logging.info("MQTT socket closed")

    def on_socket_error(self, client, userdata, exc):
        """Handle socket errors."""
        logging.error(f"MQTT socket error occurred: {str(exc)}")
        logging.exception("Full socket error details:")

    def on_connect(self, client, userdata, flags, rc):
        """Handle connection events and retry sending pending messages."""
        self.connection_attempts += 1
        self.last_connection_time = time.time()

        rc_codes = {
            0: "Connection successful",
            1: "Connection refused - incorrect protocol version",
            2: "Connection refused - invalid client identifier",
            3: "Connection refused - server unavailable",
            4: "Connection refused - bad username or password",
            5: "Connection refused - not authorized"
        }

        if rc == 0:
            self.connected = True
            self.reconnect_delay = 1
            pending_count = len(self.pending_messages)
            logging.info(f"Connected to MQTT broker after {self.connection_attempts} attempts")
            if pending_count > 0:
                logging.info(f"Attempting to send {pending_count} pending messages")

            while self.pending_messages and self.connected:
                topic, payload, qos = self.pending_messages.popleft()
                self.publish(topic, payload, qos)
        else:
            self.connected = False
            error_msg = rc_codes.get(rc, f"Unknown error code: {rc}")
            logging.error(f"MQTT connection failed: {error_msg}")
            if rc == 4:
                logging.error(f"Authentication failed for user: {self.config['user']}")

    def on_disconnect(self, client, userdata, rc):
        """Handle disconnection events with exponential backoff."""
        self.connected = False
        disconnect_time = time.time()
        session_duration = (
            round(disconnect_time - self.last_connection_time)
            if self.last_connection_time
            else 0
        )

        if rc != 0:
            logging.error(
                f"MQTT disconnected unexpectedly with code {rc} "
                f"after {session_duration} seconds"
            )
            self.reconnect_delay = min(
                self.reconnect_delay * 2, self.max_reconnect_delay
            )
            logging.info(f"Will attempt reconnection in {self.reconnect_delay} seconds")
        else:
            logging.info(
                f"MQTT disconnected normally after {session_duration} seconds"
            )

    def on_publish(self, client, userdata, mid):
        """Confirm successful message publication."""
        logging.debug(f"Message {mid} published successfully")

    def publish(self, topic, payload, qos=1):
        """Publish message with QoS and enhanced error handling."""
        try:
            if not self.connected:
                self.pending_messages.append((topic, payload, qos))
                pending_count = len(self.pending_messages)
                logging.warning(
                    f"MQTT not connected, message queued (pending: {pending_count})"
                )
                return False

            info = self.client.publish(topic, payload, qos=qos)

            if info.rc != mqtt.MQTT_ERR_SUCCESS:
                self.pending_messages.append((topic, payload, qos))
                logging.error(
                    f"Failed to publish message: {info.rc}, "
                    f"message queued (pending: {len(self.pending_messages)})"
                )
                return False

            if qos > 0:
                logging.debug(
                    f"Message queued for publishing with QoS {qos} "
                    f"(mid: {info.mid})"
                )
            return True

        except Exception as e:
            logging.error(f"MQTT Publish Error: {str(e)}")
            logging.exception("Full publish error details:")
            self.pending_messages.append((topic, payload, qos))
            return False

    def stop(self):
        """Clean shutdown of MQTT connection."""
        if self.client:
            try:
                pending_count = len(self.pending_messages)
                if pending_count > 0:
                    logging.warning(
                        f"Shutting down with {pending_count} pending messages"
                    )
                self.client.loop_stop()
                self.client.disconnect()
                logging.info("MQTT client shutdown complete")
            except Exception as e:
                logging.error(f"Error during MQTT shutdown: {str(e)}")
                logging.exception("Full shutdown error details:")

class MessageItem:
    """Message container."""

    __slots__ = (
        "timestamp",
        "message_raw",
        "body",
        "capcodes",
        "receivers",
        "is_posted"
    )

    def __init__(self, message_raw, timestamp, body, capcodes):
        self.timestamp = timestamp
        self.message_raw = message_raw
        self.body = body
        self.capcodes = capcodes
        self.receivers = " ".join(capcodes)
        self.is_posted = False

def load_config():
    """Load or create config file."""
    config = configparser.ConfigParser()
    if config.read(CFGFILE):
        return config

    config["main"] = {"debug": "true"}
    config["rtl-sdr"] = {
        "cmd": "rtl_fm -f 169.65M -M fm -s 22050 | multimon-ng -a FLEX -t raw -"
    }
    config["home-assistant"] = {
        "enabled": "true",
        "baseurl": "http://homeassistant.local:8123",
        "token": "YOUR_TOKEN_HERE",
    }
    config["mqtt"] = {
        "enabled": "true",
        "mqtt_server": "<IP_ADDRESS>",
        "mqtt_port": "1883",
        "mqtt_user": "mqttuser",
        "mqtt_password": "password",
        "mqtt_topic": "p2000",
        "mqtt_qos": "1",
    }

    with open(CFGFILE, "w") as f:
        config.write(f)
        logging.info(f"Created new config file: {CFGFILE}")
    return config

class P2000Receiver:
    def __init__(self):
        print(f"Initializing P2000 Receiver v{VERSION}...")
        self.thread_manager = ThreadManager()
        self.messages = MessageQueue(maxlen=100)
        self.config = load_config()
        self.mqtt_handler = None
        self.device_ready = threading.Event()

        # Configure logging
        self.debug = self.config.getboolean("main", "debug")
        log_level = logging.DEBUG if self.debug else logging.INFO
        logging.basicConfig(
            level=log_level,
            format="%(asctime)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S",
        )

        logging.info("Loading configuration...")

        # Load config settings
        self.rtlfm_cmd = self.config.get("rtl-sdr", "cmd")
        logging.info(f"RTL-SDR command: {self.rtlfm_cmd}")

        # Initialize Home Assistant
        self.use_hass = self.config.getboolean("home-assistant", "enabled")
        if self.use_hass:
            self.baseurl = self.config.get("home-assistant", "baseurl")
            self.token = self.config.get("home-assistant", "token")
            self.setup_home_assistant()

        # Initialize MQTT
        self.use_mqtt = self.config.getboolean("mqtt", "enabled")
        if self.use_mqtt:
            self.setup_mqtt()

        logging.info("Initialization complete.")

    def setup_home_assistant(self):
        """Initialize Home Assistant connection."""
        try:
            logging.info("Setting up Home Assistant connection...")
            self.http_session = requests.Session()
            adapter = requests.adapters.HTTPAdapter(
                pool_connections=1,
                pool_maxsize=1,
                max_retries=3,
                pool_block=False,
            )
            self.http_session.mount("http://", adapter)
            self.http_session.mount("https://", adapter)
            self.http_session.headers.update(
                {
                    "Authorization": f"Bearer {self.token}",
                    "content-type": "application/json",
                    "Connection": "keep-alive",
                }
            )
            logging.info("Home Assistant connection setup complete")
        except Exception as e:
            logging.error(f"Error setting up Home Assistant: {e}")
            self.use_hass = False

    def setup_mqtt(self):
        """Initialize MQTT connection."""
        try:
            logging.info("Setting up MQTT connection...")
            self.mqtt_config = {
                "server": self.config.get("mqtt", "mqtt_server"),
                "port": self.config.getint("mqtt", "mqtt_port"),
                "user": self.config.get("mqtt", "mqtt_user"),
                "password": self.config.get("mqtt", "mqtt_password"),
                "topic": self.config.get("mqtt", "mqtt_topic"),
                "qos": self.config.getint("mqtt", "mqtt_qos", fallback=1),
            }
            self.mqtt_handler = MQTTHandler(self.mqtt_config)
            logging.info("MQTT connection setup complete")
        except Exception as e:
            logging.error(f"Error setting up MQTT: {e}")
            self.use_mqtt = False

    def post_message(self, msg):
        """Post message to configured endpoints."""
        try:
            data = {
                "state": msg.body,
                "attributes": {
                    "time_received": msg.timestamp,
                    "receivers": msg.receivers,
                    "capcodes": msg.capcodes,
                    "raw_message": msg.message_raw,
                },
            }

            json_str = json.dumps(data)
            logging.debug(f"Prepared message data: {json_str}")

            # Post to Home Assistant
            if self.use_hass:
                try:
                    response = self.http_session.post(
                        f"{self.baseurl}/api/states/sensor.p2000",
                        data=json_str,
                        timeout=5,
                    )
                    response.raise_for_status()
                except Exception as e:
                    logging.error(f"HA Error: {e}")

            # Post to MQTT
            if self.use_mqtt and self.mqtt_handler:
                try:
                    topic = f"{self.mqtt_config['topic']}/sensor/p2000"
                    self.mqtt_handler.publish(
                        topic,
                        json_str,
                        qos=self.mqtt_config["qos"]
                    )
                except Exception as e:
                    logging.error(f"MQTT Error: {e}")

        except Exception as e:
            logging.error(f"Error in post_message: {e}")

    def process_messages(self):
        """Process messages from queue."""
        logging.info("Message processing thread started")
        startup_time = time.time()

        while self.thread_manager.running.is_set():
            try:
                msg = self.messages.get(block=True, timeout=0.1)
                if not msg.is_posted:
                    self.post_message(msg)
                    msg.is_posted = True

                    # Exit startup mode after first successful message
                    if self.messages.startup_mode:
                        self.messages.exit_startup_mode()

            except queue.Empty:
                # Check if we should exit startup mode
                if self.messages.startup_mode and time.time() - startup_time > 30:
                    self.messages.exit_startup_mode()
                continue
            except Exception as e:
                logging.error(f"Error processing message: {e}")
                time.sleep(1)

    def read_rtl_sdr(self):
        """Read and process RTL-SDR data with automatic restart capability."""
        logging.info("Starting RTL-SDR process...")

        base_delay = 1
        max_delay = 60
        current_delay = base_delay
        consecutive_failures = 0
        max_consecutive_failures = 5

        def is_usb_error(error_text):
            """Check if error indicates USB/hardware issues."""
            usb_errors = [
                "cb transfer status",
                "canceling",
                "device not found",
                "no compatible devices found",
                "USB error",
                "failed to open rtlsdr device",
                "unable to claim usb interface"
            ]
            return any(err in error_text.lower() for err in usb_errors)

        def is_startup_message(error_text):
            """Identify normal startup messages."""
            startup_msgs = [
                'multimon-ng',
                'available demodulators:',
                'found',
                'using device',
                'tuner',
                'tuned to',
                'sampling at',
                'sample rate',
                'buffer size',
                'oversampling',
                'allocating',
                'copyright',
                'output at',
                '(c)',
                'realtek',
                'rtl2838',
                'sn:',
                'by tom sailer',
                'by elias oenal'
            ]
            return any(msg in error_text.lower() for msg in startup_msgs)

        def is_device_ready(error_text):
            """Check if device is ready to receive messages."""
            ready_indicators = [
                'tuned to',
                'sampling at',
                'output at'
            ]
            return any(indicator in error_text.lower() for indicator in ready_indicators)

        while self.thread_manager.running.is_set():
            process = None
            start_time = time.time()
            usb_error_detected = False
            self.device_ready.clear()

            try:
                # Check if rtl_fm binary exists and is executable
                rtl_fm_path = self.rtlfm_cmd.split()[0]
                if not os.path.exists(rtl_fm_path) and rtl_fm_path != 'rtl_fm':
                    raise FileNotFoundError(f"RTL-SDR software not found: {rtl_fm_path}")

                # Start the RTL-SDR process
                process = subprocess.Popen(
                    self.rtlfm_cmd,
                    stdout=subprocess.PIPE,
                    stderr=subprocess.PIPE,
                    shell=True,
                    bufsize=1,
                    universal_newlines=True,
                )

                logging.info("RTL-SDR process started, waiting for device ready...")

                # Main processing loop
                while self.thread_manager.running.is_set():
                    if process.poll() is not None:
                        raise Exception(f"RTL-SDR process terminated with code {process.poll()}")

                    import select
                    reads = [process.stdout.fileno(), process.stderr.fileno()]
                    timeout = 0.1  # Reduced timeout for faster startup response

                    try:
                        readable, _, _ = select.select(reads, [], [], timeout)
                    except (select.error, ValueError) as e:
                        logging.error(f"Select error (possible USB disconnect): {e}")
                        raise Exception("USB device may have been disconnected")

                    for fd in readable:
                        if fd == process.stdout.fileno():
                            line = process.stdout.readline()
                            if line:
                                if "FLEX" in line and "ALN" in line:
                                    try:
                                        match = FLEX_PATTERN.search(line)
                                        if match:
                                            if not self.device_ready.is_set():
                                                self.device_ready.set()
                                                logging.info("First message received, device is ready")
                                            self.handle_message(line, match)
                                    except Exception as e:
                                        logging.error(f"Error parsing message: {e}")
                        elif fd == process.stderr.fileno():
                            error = process.stderr.readline()
                            if error:
                                error_text = error.strip()
                                if not error_text:
                                    continue

                                # Check for device ready state
                                if not self.device_ready.is_set() and is_device_ready(error_text):
                                    self.device_ready.set()
                                    logging.info("Device ready to receive messages")

                                if is_usb_error(error_text):
                                    logging.error(f"USB Error detected: {error_text}")
                                    usb_error_detected = True
                                    raise Exception("USB device error detected")
                                elif is_startup_message(error_text):
                                    logging.info(f"RTL-SDR: {error_text}")
                                else:
                                    logging.warning(f"RTL-SDR Error: {error_text}")

            except Exception as e:
                logging.error(f"Error in RTL-SDR process: {e}")
                consecutive_failures += 1

                # More aggressive retry for USB errors
                if usb_error_detected:
                    logging.info("USB error detected - attempting quick recovery")
                    current_delay = base_delay  # Reset delay for USB errors
                    time.sleep(2)  # Short delay for USB reconnection
                else:
                    if consecutive_failures >= max_consecutive_failures:
                        logging.warning(
                            f"RTL-SDR failed {consecutive_failures} times in rapid succession. "
                            "There might be a hardware or system issue."
                        )

                    if time.time() - start_time < 10:
                        current_delay = min(current_delay * 2, max_delay)

                    logging.info(f"Waiting {current_delay} seconds before restarting RTL-SDR...")

                    # Sleep but respect shutdown signal
                    sleep_start = time.time()
                    while (time.time() - sleep_start < current_delay and
                           self.thread_manager.running.is_set()):
                        time.sleep(1)

            finally:
                if process:
                    try:
                        process.kill()
                        process.wait(timeout=5)
                    except Exception as e:
                        logging.error(f"Error cleaning up RTL-SDR process: {e}")

                if self.thread_manager.running.is_set():
                    logging.info("RTL-SDR process ended, preparing for restart...")

    def handle_message(self, line, match):
        """Process and queue a new message."""
        try:
            timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
            capcodes = match.group(4).strip().split()
            message = match.group(5).strip()

            if message and capcodes:
                msg = MessageItem(
                    message_raw=line.strip(),
                    timestamp=timestamp,
                    body=message,
                    capcodes=capcodes,
                )
                self.messages.put(msg, block=False)
                print(f"[{timestamp}] New message: {message}")
        except Exception as e:
            logging.error(f"Error handling message: {e}")

    def run(self):
        """Start the receiver."""
        logging.info(f"P2000 Receiver v{VERSION} starting...")

        try:
            # Start message processing thread
            self.thread_manager.start_thread("processor", self.process_messages)

            # Start RTL-SDR reading in separate thread
            self.thread_manager.start_thread("rtl-sdr", self.read_rtl_sdr)

            # Wait for shutdown signal
            try:
                while self.thread_manager.running.is_set():
                    time.sleep(1)
            except KeyboardInterrupt:
                logging.info("Shutdown signal received")

        finally:
            # Cleanup and stop all threads
            self.thread_manager.stop_all(timeout=5)
            if self.mqtt_handler:
                self.mqtt_handler.stop()
            logging.info("Receiver stopped")

def main():
    """Main entry point with error handling."""
    try:
        # Set up signal handlers
        def signal_handler(signum, frame):
            logging.info(f"Received signal {signum}")
            if receiver:
                receiver.thread_manager.running.clear()

        import signal

        signal.signal(signal.SIGINT, signal_handler)
        signal.signal(signal.SIGTERM, signal_handler)

        # Create and run receiver
        receiver = P2000Receiver()
        receiver.run()

    except KeyboardInterrupt:
        print("\nShutting down...")
    except Exception as e:
        print(f"Fatal error: {str(e)}")
        logging.exception("Fatal error occurred")
        sys.exit(1)

if __name__ == "__main__":
    main()


When adding remappings.txt, forge build throws an error

Component
Forge
Have you ensured that all of these are up to date?

[X] Foundry
[ ] Foundryup

What version of Foundry are you on?
forge 0.2.0 (3480a8d 2022-04-03T11:39:36.163735900+00:00)
What command(s) is the bug in?
forge build
Operating System
Windows
Describe the bug
After running
forge init hello
cd hello
forge remappings > remappings.txt

, forge build throws the following error ("stream did not contain valid UTF-8"):
The application panicked (crashed).
Message:  failed to extract from provider: Error { tag: Tag(Default, 7), profile: Some(Profile(Uncased { string: "default" })), metadata: Some(Metadata { name: "Remapping Provider", source: None, provide_location: Some(Location { file: "config\\src\\lib.rs", line: 808, col: 29 }), interpolater:  }), path: [], kind: Message("stream did not contain valid UTF-8"), prev: None }
Location: config\src\lib.rs:286

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   1: hid_write<unknown>
      at <unknown source file>:<unknown line>
   2: hid_write<unknown>
      at <unknown source file>:<unknown line>
   3: hid_write<unknown>
      at <unknown source file>:<unknown line>
   4: BaseThreadInitThunk<unknown>
      at <unknown source file>:<unknown line>
   5: RtlUserThreadStart<unknown>
      at <unknown source file>:<unknown line>

If I delete the remappings.txt the error goes away and the contract compiles.
remappings.txt file:
  Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000: FF FE 64 00 73 00 2D 00 74 00 65 00 73 00 74 00    .~d.s.-.t.e.s.t.
00000010: 2F 00 3D 00 6C 00 69 00 62 00 5C 00 64 00 73 00    /.=.l.i.b.\.d.s.
00000020: 2D 00 74 00 65 00 73 00 74 00 5C 00 73 00 72 00    -.t.e.s.t.\.s.r.
00000030: 63 00 2F 00 0D 00 0A 00                            c./.....

it appears your path is not utf8 conform, which is a windows thing..
try deleting the line and replacing it with
ds-test/=lib/ds-test/src/

changing it to ds-test/=lib/ds-test/src/
  Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000: FF FE 64 00 73 00 2D 00 74 00 65 00 73 00 74 00    .~d.s.-.t.e.s.t.
00000010: 2F 00 3D 00 6C 00 69 00 62 00 5C 00 64 00 73 00    /.=.l.i.b.\.d.s.
00000020: 2D 00 74 00 65 00 73 00 74 00 5C 00 73 00 72 00    -.t.e.s.t.\.s.r.
00000030: 63 00 2F 00 0D 00 0A 00                            c./.....

produces the same error.

was just about to write that,
just noticed all the null bytes.
rust strings are utf8

not a windows expert but interesting that powershell, pipes content as utf-16


Circular import or some other import issue

At some point during the recent updates I have broken the queenbee cli import process. queenbee --help returns an error for importing the Operator class. I need to go backwards and see when and why this has happened.

fixed via https://github.com/ladybug-tools/queenbee/pull/157/commits/02b0b6a232fff5bc2a27549606b342e01def080a


Extra tools needed for arm64

I'm running Linux on arm64. I had to add
RUN apk add --no-cache python2 g++ make
before
WORKDIR /app
in my Dockerfile

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: cddd74b1-e0f1-1cae-e92d-c7c2b81e77e3
Version Independent ID: f853e977-81c2-64d6-682e-1e712344cbad
Content: Tutorial: Get started with Docker apps in Visual Studio Code
Content Source: docker/tutorials/docker-tutorial.md
Product: vs-code
GitHub Login: @ucheNkadiCode
Microsoft Alias: uchen

@ghogen -- Gordon, please review this addition in private PR 10582.


can't figure out how to disable properly shortlink, canonical and robots

I tried to enable shortink, canonical and robots just for the admin.
In app/config/extensions/seo.bodenotter.yml, the default proposition is:
allow:
    shortlink: [chief-editor, admin, developper, root]
    canonical: [chief-editor, admin, developper, root]
    robots: [chief-editor, admin, developper, root]

I uncommented these lines modifying it like this:
allow:
    shortlink: [admin]
    canonical: [admin]
    robots: [admin]

But it doesn't work.
In vendor/bobdenotter/seo/templates/_seoextension_field.twig, I noticed lines 109, 123, 137 that the variables are set to seoconfig.allowed.options|default(true)
The only way I succeeded to disable canonical, robots and shortlink was to hardcode this file, deleting the default option, resulting in seoconfig.allowed.options instead of seoconfig.allowed.options|default(true) for each lines cited above.
And modifying my seo.bodenotter.yml, I did that:
allowed:
    shortlink: false
    canonical: false
    robots: false

It effectively disabled these 3 options but as I'm not a fan about hardcoding a vendor file because of the possible override after an update, I want to know if there is a better way to do that. And how is it possible to enable these options just for the admin?
Thanks for your patience and I apologize in advance if my English is incorrect, hoping you guess the meaning of my question.

Looks like the templates and settings from #8 survived recent updates to the extension, but not the config array creation part.
The current project I'm on would benefit from the role based permissions, so if I manage to get the solution right I'll submit a pull request with the change.
Hope that helps.


Need non-wear time for user defined day.

My group is working with GGIR to reduce data from GENEActive accelerometers. We get through parts 1 and 2. The problem is that we are defining our days as between 5:00 and 23:00 and need the count of valid wear hours or non-wear hours for that time for each day. We are going to use the weighting method in a recent Xu et al paper (https://www.ncbi.nlm.nih.gov/pubmed/27405327) to address non-wear. From part 2 we get the number of valid minutes per day of wear, but it is per 24 hour day and the use of qwindow = c(5,23) does not give us the wear or non-wear time in our desired window. There must be a way to do this?
Thanks, Lee (University of New Mexico)

Hi Lee, The paper you are referring to is not open access, by which i am not able to read it.
Please note that GGIR already imputes non-wear time by default as described in our publications: It takes the average of all valid data (monitor worn) on similar time points on other days of the measurement. Effectively this is a weighted imputation of average daily acceleration, because days with more valid data will have stronger influence on the overal mean.
Note that all intermediate stages of information are directly accessible in the milestone data stored in RData files, which include epoch by epoch indicators of whether the accelerometer was worn. With this you would be able to design your own imputation scheme.
From the abstract it is not clear to me what the new method does, whether the authors provide any evidence that the method is superior to existing methods like the one implemented in GGIR, and whether it is truly any different compared with what already exists but just using different terminology.

Thanks for the quick response Vincent. OK, so it looks like I need to mine the RData files to find this info? Will work on that. Is it also possible to get it this information in Part 5 if we specify the day as 5:00 to 23:00 in Part 4? Also, we really appreciate the work you have done on GGIR, your non-wear scores appear to be especially compelling, way better than what my teams have used before to define non-wear!
My note, as a methodologist who has spent a lot of time dealing with missing data and with missing accelerometer data, is that simply replacing non-wear points with points from similar days is problematic for all of the reasons that using single imputation is problematic. The easiest example is this: if someone just meets your minimum wear criteria (say 3 days), you then simply use their values over those 3 days. For someone who wore the accelerometer for the entire week, there is no imputation of missing data. Thus, in your analyses you are treating these two individuals as if their values are equal when infact you have much more information for the person who wore the device for the entire week. The repeated measures weighting approach will treat those two subjects very differently as parameter estimates at level-2 are adjusted for the precision of the level-1 data (how many data points each person has).
I guess an important point here is that the weighted approach doesn't use week level means, instead it is a repeated measures (multilevel) model which is based on day-level data. This is why it ends up differing substantially from your imputation approach, because the method allows an individual who has 3 days of data to be included but gives more weight to the person with 7 days of data. In fact, there is no problem including individuals with 1 day of data, we aren't planning to drop anyone based on minimum wear times.
It sounds semantic, but failing to address the uncertainty caused by non-wear has two problems: 1) you have a lot more unreliability for people with lots of non-wear, that should decrease the size of regression weights predicting PA; and 2) your analyses believe that you have more information than you actually do (since you are filling in missing values with 1 value and thus your standard errors will be too small. Of course, the effects of this depend a lot on the particular situation, but in the paper referenced above you can see the impact of the 'naive approach' on the precision of parameter estimates. I guess what I'm arguing is that the naive approach (just using data which meets criteria) and filling in values from similar times of day are very similar from a statistical perspective and less than optimal. We have previously spent a lot of time using multiple imputation and I'm not sure it is worth the time cost, but using a weighted longitudinal approach is quite simple and has the advantage of being a principaled statistical approach for addressing non-wear.
Best, Lee

Thanks for explaining Lee, it sounds like I will have to carefully read that paper then! I have a bit of experience with multi-level modelling, but don't consider myself an expert. If you think it is worth expanding GGIR with the repeated measures weighting approach then I am happy to facilitate.
You will need to look out for object rout in the part2 output see also the tutorial: https://cran.r-project.org/web/packages/GGIR/GGIR.pdf As the documentation says rout holds binary indicators per timestamp of whether the data can be trusted or not. Let me know if anything is unclear.
Those timestamps should then match with the timestamps in part5 output.
I agree that it may be easier to have time series including non-wear indicators stored as part of the part5 output, together with all the level labels or to have some kind of export function that exports all these time series to csv files. I just haven't got round building something like that yet.

Correction: rout does not have timestamps, but is aligned with IMP$metalong the less dense indicator of non-wear time (by default 15 minutes but adjustable). So, first you would have to match rout to the timestamps of the metalong dataframe inside the IMP or M object.

Thanks for this Vincent,
we will work on it to see what we can get working with our data. The nice thing about this approach is that once the data is in the right format the model is quite simple. I have a post doc working on this in another project and our preliminary results look pretty good, some example code is below, once we figured out the right way to include the weights it is quite painless to estimate. The challenge is to get the data output in the right format, we need a 'stacked' file with one row per time period of interest (if modeling PA per day, each person needs a row for each day they were observed). We can get this already from GGIR, all we need to do is add the proportion of non-wear time associated with each row. Once we have something I'll try to remember to post code here to help you and anyone else trying to do this. Unfortunately I don't have enough programming time for this project, what else is new?
By the way, if interested, here is a link to a paper that a former grad student did with some other more interesting uses of multilevel analysis of accelerometry data (https://www.researchgate.net/publication/265388428_A_multilevel_approach_to_examining_time-specific_effects_in_accelerometer-assessed_physical_activity). Basically, she showed that our power to look at intervention effects at specific time points was far greater than for looking at intervention effects on average, there are lots of applications beyond RCTs.
Best!  Lee
Multilevel model:
MVPA.C<-lme(SQ.C.MVPA ~ income.c + Married + Child_male, random = ~1|FID,  data = PA.BL, na.action = na.exclude, method = 'REML')
#Updating with the sampling weights:
MVPA.C.Weight<-update(MVPA.C, data = PA.BL, method='REML', weights=varFixed(~1/(1-C.Prop)))

Thanks Lee,
It may be worth adding that I did use multi-level modelling to account for person/night level missing data in one study: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0142533. So, I worked with the night specific sleep estimates (27 thousand nights in 4000 individuals), and used multi-level modelling to account for the variation between individuals and within individuals (between nights). Before running the analyses I omitted all days with insufficient monitor wear time (monitor worn for < 66% of the day). This then allowed the multi-level model to put more weight on individuals with more valid days of data. It is better than standard single level regression, but I agree that this approach indeed does not account well for within day data availability. So, in my case a day with 22 hours of data + 2 imputed hours weighted equal compared with a day with 24 hours of data. I can see why that is not ideal and look forward seeing your results!


Setting RetryAttempts to 0, always throws an exception

Expected behavior
I can set retryAttempts to 0, when I don't want to retry the command.
This was working in version 3.20.0, but seems to break after upgrading to 3.23.1
Actual behavior
if the retryAttempts is set to 0 in version 3.20.0 it works fine.
if the retryAttempts is set to 0 in version 3.23.1, then it throws an exception:
java.util.concurrent.CompletableFuture@29c53c4f[Completed exceptionally: org.redisson.client.RedisNodeNotFoundException: Node for slot: 0 hasn't been discovered yet. Check cluster slots coverage using CLUSTER NODES command. Increase value of retryAttempts and/or retryInterval settings.

stack trace:
org.redisson.client.RedisNodeNotFoundException: Node for slot: 0 hasn't been discovered yet. Check cluster slots coverage using CLUSTER NODES command. Increase value of retryAttempts and/or retryInterval settings.
	at org.redisson.connection.ServiceManager.createNodeNotFoundException(ServiceManager.java:290)
	at org.redisson.command.RedisExecutor.connectionWriteOp(RedisExecutor.java:753)
	at org.redisson.command.RedisExecutor.getConnection(RedisExecutor.java:672)
	at org.redisson.command.RedisExecutor.execute(RedisExecutor.java:128)
	at org.redisson.command.CommandAsyncService.async(CommandAsyncService.java:535)
	at org.redisson.command.CommandAsyncService.evalAsync(CommandAsyncService.java:488)
	at org.redisson.command.CommandAsyncService.evalWriteAsync(CommandAsyncService.java:364)
	at org.redisson.RedissonMap.putOperationAsync(RedissonMap.java:1345)
	at org.redisson.RedissonMap.putAsync(RedissonMap.java:1335)
	at org.redisson.RedissonMap.put(RedissonMap.java:659)

Steps to reproduce or test case
use a simple cmd like :
redissonClient.getBucket(key, StringCodec.INSTANCE).setAsync(value, expirationInSeconds, TimeUnit.SECONDS);

Redis version
AWS Elasticache with Redis engine 7.0.7
Redisson version
3.23.1
Redisson configuration
  private Config buildRedissonConfiguration() {
        final Config config = new Config();
        config.setEventLoopGroup(eventLoopGroup);
        config.useSingleServer()
              .setClientName(clientConfiguration.getClientName())
              .setAddress(clientConfiguration.getHostAndPort())
              .setConnectionPoolSize(100)
              .setConnectionMinimumIdleSize(50)
              .setConnectTimeout(1000)
              .setPingConnectionInterval(0)
              .setRetryAttempts(0)
              .setTimeout(1000)
              .setIdleConnectionTimeout(30_000);

        return config;
    }

Fixed. Thanks for report


Use CLI magic

depends on release of https://github.com/fsprojects/Paket/pull/2486

Note: this should only be merged when paket 5.5 is released. and we should not merge the paket config file

OK changed my mind - I will merge this now and it will use the paket 5.5 rc1. After release we go back on stable channel. But I really want people to test it


Do not show Hexa icon for F&F on Account Tiles

In the current release it shows a Hexa icon for each F&F on an account. We do not need this.

For future reference: good discussion here about how we decided to handle this.

-Verified this issue on staging app version 1.4.0(184)


[New game]:  Rock Paper Scissor

🎮 Game Request
A rock paper scissor game, one can play with computer
Point down the features
A basic rock paper scissor game where player will have choice of points till he/she wants to play.
Select program in which you are contributing
GSSoC23
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @YaShRaJ0P ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇


does not combine sysctl files for version higher than 0.9.0

Cookbook version
0.10.1
Scenario:
attributes/default.rb file is missing since version 0.10.0, which exists on previous versions
Scenario:
using version higher than 0.9.0 does not combine sysctl files
Expected Result:
to combine sysctl files into a single file
Actual Result:
does not combine sysctl files into a single file

We are having the same issue.  We cannot go past 0.9.0 because it keeps failing.
================================================================================
Recipe Compile Error in /var/chef/cache/cookbooks/nfs/attributes/default.rb
================================================================================

Chef::Exceptions::AttributeNotFound
-----------------------------------
could not find filename for attribute default in cookbook sysctl

Cookbook Trace:
---------------
  /var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:389:in `resolve_attribute'
  /var/chef/cache/cookbooks/nfs/attributes/default.rb:20:in `from_file'

Relevant File Content:
----------------------
/var/chef/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/run_context.rb:

382:      # @raise [Chef::Exceptions::AttributeNotFound] If the attribute file could not be found.
383:      #
384:      def resolve_attribute(cookbook_name, attr_file_name)
385:        cookbook = cookbook_collection[cookbook_name]
386:        raise Chef::Exceptions::CookbookNotFound, "could not find cookbook #{cookbook_name} while loading attribute #{name}" unless cookbook
387:
388:        attribute_filename = cookbook.attribute_filenames_by_short_filename[attr_file_name]
389>>       raise Chef::Exceptions::AttributeNotFound, "could not find filename for attribute #{attr_file_name} in cookbook #{cookbook_name}" unless attribute_filename
390:
391:        attribute_filename
392:      end
393:
394:      #
395:      # A list of all recipes that have been loaded.
396:      #
397:      # This is stored internally as a Hash, so ordering is predictable.
398:      #

Platform:
---------
x86_64-linux

Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 06 seconds

@damacus: i would be happy to help you in anyway i can, though i do not have any development and testing environment. how do you think i could help?

Hey @y0y0z we can do one of two things (or both).

Revert the removal of attributes/default.rb
Ignore the breaking change and look at moving to the long term goal of making custom resources work.

@damacus: i do not have any development environment. i can fork, do the modifications, push the changes, and submit a pull-request. though, i cannot clone the repository on my local machine. to address your first bullet, i think executing git checkout bd9cc0f attributes/default.rb should do the trick. yet, i am unsure whether it will resolve the issue as i have no way of testing it.
sorry i couldn't do more right now :(

Do you have a copy of ChefDK + vagrant + virtualbox or something similar?

@damacus: if i had it, i would have create pull request

my version is still locked on 0.9.0


Missing remove button

Is the remove button missing on the inventory page for everyone else?

Yeah, I removed it cause we really don't need that feature
On Mar 25, 2015 8:29 AM, "m-nguyen"<EMAIL_ADDRESS>wrote:

Is the remove button missing on the inventory page for everyone else?
—
Reply to this email directly or view it on GitHub
https://github.com/cphazen/Inventory-System/issues/22.

What if someone accidentally adds an item?

That's true, sorry about that, didn't really think about that
On Mar 25, 2015 9:50 AM, "Courtney Hazen"<EMAIL_ADDRESS>wrote:

What if someone accidentally adds an item?
—
Reply to this email directly or view it on GitHub
https://github.com/cphazen/Inventory-System/issues/22#issuecomment-86034755
.


Remove repo name

Now that repo is the new column that is being used in all of the tables, the old repo_name and repo_owner columns can be dropped for good.
QA: I'll do it since the setup is annoying.
Closes: https://github.com/iFixit/pulldasher/issues/79

dev_block :+1: on https://github.com/iFixit/pulldasher/pull/82 being merged

#82 is merged now — unblock?

Yeah, time to get the merge conflicts fixed.

Maybe rebase so there is only a few commits?

I'll get to this soon, it's just not a critical pull compared to the other stuff that needs to be done at the moment.

CR :zap:

Pøke

This might need a rebase... the commit history looks bigger than it should. Probably should only have the last 5 commits or so.

un_dev_block :+1: did the rebase

@MooreGuy I can QA this with you.

CR :+1:
I was going to say, perhaps move this migration: https://github.com/iFixit/pulldasher/pull/83/commits/385e52537ab16abac918ea6b7b6d315b6eb35c84
to a separate pull, but this is pulldasher, so it's probably fine. I'm going to take a stab in the dark and guess that migrations get ran on startup, right? If not, then it seems like that migration could be a break change under a live running pulldasher. Nothing to block on.

dev_block 👍  on a comment.

@djmetzle The migrations for pulldasher need to be run manually. How we have been doing the migrations is run the migration and then restart pulldasher so it is running the new code. Doing it this way won't break anything.

Actually I take that back, I didn't realize there were two migrations @djmetzle I'm going to split them up.

@MooreGuy I wouldn't worry too much then. You can just stop the pulldasher service and restart it after the migrations. I was just concerned that if pulldasher was running when you ran it, that migration could break stuff.

dev_block on this having the second query/schema change removed

un_dev_block :+1:

QA :+1: I checked the frozen pulls link, the repo links, and CR'd QA'd stuff

CR 👍

CR 👍

CR :+1:

Guy do you want to give this a QA?

CR 👍

@mlahargou You should be set up to test it now, I updated QA instructions to follow. There isn't a whole lot in pulldasher, so it's pretty reasonable to skim through most of its functionality.

Actually yeah, I already went through this, I'll do it

@MooreGuy I'm taking the external block off. Feel free to QA it if you want, but it seems like someone else can do this.

I got really frustrated with dealing with two pull requests that both need to be merged to QA, so this one plus more is now in: https://github.com/iFixit/pulldasher/pull/109


See for example:

Fedora CI log
Testing Farm log

This was supposed to be fixed with #786
@csomh ideas?

@jpopelka how do these environments look like? how are they set up? When are they recreated? I cannot really tell it from the logs. Would be nice to know if there is any $HOME/.gitconfig there.

I don't have answers to those questions.


lunar-client: 3.2.7 -> 3.2.9

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for lunar-client is: Free Minecraft client with mods, cosmetics, and performance boost.
meta.homepage for lunar-client is: https://www.lunarclient.com/
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 3.2.9 with grep in /nix/store/jj30yxs698s679ns1z5sndbx9dl993ds-lunar-client-3.2.9
found 3.2.9 in filename of file in /nix/store/jj30yxs698s679ns1z5sndbx9dl993ds-lunar-client-3.2.9

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

lunar-client

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/jj30yxs698s679ns1z5sndbx9dl993ds-lunar-client-3.2.9 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A lunar-client https://github.com/r-ryantm/nixpkgs/archive/9452c6886171d0c6d13b8492ca00b75ca37a97f6.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/9452c6886171d0c6d13b8492ca00b75ca37a97f6#lunar-client

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/jj30yxs698s679ns1z5sndbx9dl993ds-lunar-client-3.2.9
ls -la /nix/store/jj30yxs698s679ns1z5sndbx9dl993ds-lunar-client-3.2.9/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

lunar-client

Maintainer pings
cc @zyansheep @Technical27 @surfaceflinger for testing.

Result of nixpkgs-review pr 314685 run on x86_64-linux 1

1 package built:

lunar-client


Make .etl output as optional

The .etl file is large, and we do not want it most of times. For example, when the perf run is nightly, and we just want the xml output to analyze the trend.

The data in the xml output comes from the .etl file, so we do need to generate it.  However, if you don't want to keep it, you can go ahead and delete it after the run.  Our team plans to keep the ETL files for as long as we have available storage, as they're very useful for diagnosing the causes of regressions, which can sometimes be hard to reproduce in another environment.


[Bug]: The DataflowRunner behavior is changed when removing the runner v1 code

What happened?
The attached notebook is broken now due to #27196.
Dataflow_Word_Count.ipynb.zip
The issue is p = beam.Pipeline(InteractiveRunner()) is used to create the pipeline but later when switching to DataflowRunner with DataflowRunner().run_pipeline(p, options=options), DataflowRunner takes the Dataflow job as the legacy runner and disables the Runner V2.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components

[X] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

@robertwb FYI

DataflowRunner().run_pipeline(p, options=options) should work just as well as before. Could you clarify exactly what is going wrong?

Repro:
import argparse
import logging

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.runners import DataflowRunner
from apache_beam.runners.interactive.interactive_runner import InteractiveRunner

def run(argv=None):
  parser = argparse.ArgumentParser()
  options = PipelineOptions()
  p = beam.Pipeline(InteractiveRunner())
  p | beam.Create([1])
  beam.runners.DataflowRunner().run_pipeline(p, options=options).wait_until_finish()

if __name__ == '__main__':
  logging.getLogger().setLevel(logging.INFO)
  run()

python pipeline.py  --project google.com:clouddfe --temp_location=gs://clouddfe-valentyn --staging_location=gs://clouddfe-valentyn --region us-central1

fails with
ERROR:apache_beam.runners.dataflow.dataflow_runner:2023-09-15T18:10:32.288Z: JOB_MESSAGE_ERROR: Runnable workflow has no steps specified.
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2023-09-15_11_10_28-12637281573769828865 is in state JOB_STATE_FAILED

I could not figure out what cause this issue. I am wondering whether we need call _check_and_add_missing_options somewhere.

This boils down to https://s.apache.org/no-beam-pipeline :).
It looks like manually setting these flags after construction may be sufficient. Right now we're hijacking the apply() on every PTransform to set them, which would be good to see if we can avoid. It should also be possible to fix this service-side so the client doesn't need to pass these arguments.


Invertibleness of generators not preserved by export/import

If you make a generator invertible, it will forget that it was invertible when reimported.

Yes I have been meaning to fix that. It's because we don't serialise any of the generator options (because on master that would require versioning and I didn't feel like doing it if we're going to break backwards compatibility anyway when merge labelled in). I will fix for the labelled branch.


wrapLines breaks some code highlighting; fix included

Where wrapLines splits the code content on \n and wraps it in <div>s if the highlighter opens a <span> on one line and doesn't close it until the next line the browser will auto-close the span in the first div and the second line won't inherit the styles. This causes code formatting to appear broken - I thought it was an issue with highlight.js so switched out for prism as seen in #444 but it persisted.
https://github.com/gnab/remark/blob/868781c84457bd1f0e0c1ee657423136fdbe3c75/src/remark/views/slideView.js#L281-L292
I've managed to work around the issue by pre-empting this in my highlightBlock override and closing spans out at the end of a line and re-opening them at the beginning of the next line. ES6 code to achieve this is here:
function parseLanguage(lang) {
  return {
    js: 'jsx',
  }[lang] || lang;
}
remark.highlighter.engine.highlightBlock = block => {
  const language = parseLanguage(block.className.split(" ")[0]);
  const prismLang = Prism.languages[language];
  if (prismLang) {
    block.parentNode.className = `${block.parentNode.className} language-${language}`;
    const html = Prism.highlight(block.textContent, prismLang);

    const lines = html.split(`\n`);
    let currentSpan = null;
    for (var i = 0; i < lines.length; i++) {
      let line = lines[i];
      if (currentSpan) {
        line = currentSpan + line;
        currentSpan = null;
      }
      const openTags = [];
      const re = /(<span[^>]*>|<\/span>)/gi;
      let matches;
      while ((matches = re.exec(line)) != null) {
        const tag = matches[1];
        if (tag[1] === '/') {
          openTags.pop();
        } else {
          openTags.push(tag);
        }
      }
      currentSpan = openTags.join('');
      line = line + ('</' + 'span>').repeat(openTags.length);
      lines[i] = line;
    }
    block.innerHTML = lines.join('\n');

  } else {
    console.warn(`Language '${language}' not supported?`)
  }
};

Perhaps you'd consider incorporating the part from const lines = html.split... to lines.join('\n') into the wrapLines function directly?

(I absolutely love this presentation framework by the way, have been using it for years and extoll the virtues of it everywhere I present ❤  - I've even got an awesome setup now with Browsersync where when you save your markdown file the browser automatically replaces all the slides instantly and jumps to the first edited slide which has increased my productivity significantly!)

For those interested in my workflow, check out this script:
https://github.com/GraphQLTraining/lightweight-graphql-react/blob/gh-pages/browsersync
And the corresponding main function in the index.html file:
https://github.com/GraphQLTraining/lightweight-graphql-react/blob/gh-pages/index.html

Thank you so much for sharing your setup @benjie - I am literally 2x more productive with it. @gnab How do you feel about adding this to remark as a default dev experience? It's sort of magical.

Oh, this is great! I wish I didn't have to hack remark to make it highlight 'jsx' properly. I would settle for an option to disable "highlight.js" and the remark code blocks altogether so I could simply link in my preferred highlighter (prism.js) for my   tags. Or something like that.

I'm using your changes to (remark.prism.js) to have the syntax highlight and it works fine. Thank you.
Do you know if it will be possible to have some plugins like line-numbers working too ?
Thank you.


README.md: list selling points of regl.

Selling points of regl are now explicitly listed in the README

This takes care of issue #84


rive-renderer integration

Since rive-renderer is now open-source, are there any plans to integrate it with rive-rs? Any updates on the timeline would be highly appreciated!

I cannot give you a timeline, but I can tell you that this requires some planning before. Integrating our renderer with wgpu will likely require exposing some functionality on the wgpu side.

Hi, are there any updates on this?

From my understanding, Dragos doesn't work at Rive anymore and this was very much a side project within Rive. It seems unlikely Rust bindings or the Bevy integration will be continued.


Does it support Windows?

Currently on windows it prints the ASCII of that image:

Does it support Windows?

Maybe? It depends on whether cmd.exe supports ANSI escapes, which I'm unaware of.

Currently on windows it prints the ASCII of that image:

This is a genuine bug! It's defaulting to the iTerm2 mode — I can tell because of the ]1337; followed by the base64 dump of the image. This should not happen; it should maybe try the ANSI 8 colour output first (most compatible) before trying the more advanced modes (e.g., iTerm2 mode).

@Jeff-Tian how did you build it with target on Windows? I can't even execute the ./configure command


Upgrade clusters to v0.31.0

Task
Upgrade CAPA clusters to cluster-aws v0.31.0 and default-apps-aws v0.25.0.
Clusters:

[x] anzu (schema migration is required)
[x] goat
[x] golem
[x] grizzly
[x] thor
[ ] thunder
[ ] thunder / dev00

thunder and dev00 can only be upgraded during the maintenance window.

We need to wait for the next maintenance window.

thunder and dev00 will be upgraded as part of this https://github.com/giantswarm/roadmap/issues/2421


Installation changes almost nothing

Installation changes almost nothing
Hello, I'm under Fedora 33 and bdiscordctl doesn't work: it shows me that the installation succeeded, but the only traces of BbetterDiscord that I have are:

a smaller interface (I'm at 125% normally, against 100% with betterdiscord)

This extra separator:

The banner proposing me to install discord

I'm mute and I can't unmute myself

And a different footer:

Without BD:

With BD:

I quickly understood that the only thing that the installation changed was that the web version was "emulated"...
I installed discord via dnf, running the command with sudo just doesn't work.
I tried reinstalling discord via snap, same result. I confess I don't know what to do, not being an expert myself...
Here's my command and its result :
[Mathis@laptop-de-mathis ~]$ betterdiscordctl install
Cloning https://github.com/rauenzi/BetterDiscordApp...
Clonage dans '/home/Mathis/.local/share/betterdiscordctl/bd/0'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 13 (delta 0), reused 6 (delta 0), pack-reused 0
Réception d'objets: 100% (13/13), 5.55 Kio | 5.55 Mio/s, fait.
Installed. (Restart Discord if necessary.)

As you can see, absolutely everything went well.
Any help is welcome!

Duplicate of #96.


Update data-classification-activity-explorer-available-events.md

@chrfox  win 32 appeared more than once. I removed all of them now

@JohanFreelancer9 - kindly do the copy editor review for this PR. Thanks


Feature/hotfix markdown fences

Address issues from closed PR

@Flamenco Could you add link to the issue that was fixed?

fixes https://github.com/getgrav/grav/issues/1840

Thank you.

I think you forgot to commit the language file?  Should update the languages/en.yaml file to get the base language strings in there.

I put the strings in the admin plugin...
https://github.com/getgrav/grav-plugin-admin/pull/1384

https://github.com/getgrav/grav-plugin-admin/pull/1385

BTW @Flamenco does this still work with latest Parsedown updates? Seems the 1.7.0+ releases have broken some more advanced markdown things.

It should.  It just escapes things so they are not treated as markdown.  And then puts them back after calling the pre-processor.


a324451936699891 Add scale 2 0 to refactoring

This change is

Coverage increased (+0.05%) to 50.047% when pulling 33040df0e381fd09464d41e124cb0731bcc1fa08 on annawzz:a324451936699891_scale_2_0_on_refactoring into 80e79b41fe7b1e26c9b75c394ff4c573e5d99b6f on openprocurement:a258001251842882_refactoring.

@ktarasz глянь ще ти


Support Hooking C Functions by Name Similar to fishhook

Title:
Description:
Hi Antibioticss,
I would like to request a new feature that enables hooking C functions by their names, similar to the way fishhook works. The ability to hook functions by name is incredibly useful for intercepting and modifying function behaviors dynamically. Currently, the project lacks this feature, which limits its usability for scenarios requiring fine-grained control over C function calls.
Feature Details:

Hook by Function Name: Add support for hooking C functions by providing their names as strings.
Compatibility: Ensure compatibility with common C standard library functions, such as malloc, free, etc.
Dynamic Resolution: Enable the hook to dynamically resolve symbols at runtime, even if the function is defined in a shared library.

I just saw the github repository, and the latest tinyhook seems to have solved this problem, but it would be nice if a get Image VM Addr Slide Index was also added.