Problem with LetsEncrypt

As an addition to my previous issue https://github.com/basecamp/kamal/issues/897 there one more problem with permissions. If you have user other than root in your deploy.yml file
ssh:
  user: alec

and traefik is configured to use letsencrypt
traefik:
  options:
    network: "private"
    publish:
      - "443:443"
    volume:
      - "/letsencrypt/acme.json:/letsencrypt/acme.json" # <------
  args:
    accesslog: true
    log.level: DEBUG
    accesslog.format: json

    entryPoints.web.address: ":80"
    entryPoints.websecure.address: ":443"

    entryPoints.web.http.redirections.entryPoint.to: websecure
    entryPoints.web.http.redirections.entryPoint.scheme: https
    entryPoints.web.http.redirections.entrypoint.permanent: true

    certificatesResolvers.letsencrypt.acme.email<EMAIL_ADDRESS>    certificatesResolvers.letsencrypt.acme.storage: "/letsencrypt/acme.json" # <------
    certificatesResolvers.letsencrypt.acme.httpchallenge: true
    certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint: web

you'll be unable to store cert locally, because of insufficient permissions (/letsencrypt will be owned by root).
I've tried to follow recommendations from this issue https://github.com/basecamp/kamal/issues/667  but it doesn't work. I've tried

chown on /letsencrypt
create ./letsencrypt in home directory + configure deploy.yml

Any idea how to fix it?

This should be resolved in Kamal 2. kamal-proxy handles the Let's Encrypt challenge and storing the certificate for you.


Add constructor based components support

Currently all components require to have a initialize. Which is called when the component is created. With the todays usage of ES6 classes this is not longer the way to go and we should  calling the constructor instead and give the element directly to it.
This helps to use TypeScript for web-js components as there all properties can be defined in the constructor:
export default class {
    constructor(el, options) {
        this.text = options.text;
        el.onclick = this.say;
    }

    say() {
        alert(this.text);
    }
}

Also a lot easier function based components can be created:
export default function(el, options) {
    el.onclick = function() {
        alert(options.text);
    };
}

TODO

[ ] Upgrade existing components

I like this change. THX 👍 🚀


created dashboard overview document

Created dashboard overview page

Looks good to me. Just one comment:
@josep-fl would be great to use commitizen style for commits to keep consistency


Hive's rand for sparklyr syntax

Hello,
Could hive's rand function be supported in sparklyr syntax?  I'd like to generate a random number based on a column input.  In hive/spark-sql, I'd write:
select myVal, rand(myVal) as myRand from myTable;
Is there some other way of achieving the same result?
Thank you,
Nathan

you can use sparklyr::spark_apply to generate random number by R native function like dnorm();
or register your dataframe and run sparksql

You can use any Hive functions inside mutate. For example
df <- copy_to(sc, data.frame(a=c(1,2,3)))
df %>% mutate(random=rand()) %>% head()
# # Source:   lazy query [?? x 2]
# # Database: spark_connection
#       a random
#   <dbl>  <dbl>
# 1  1.00  0.213
# 2  2.00  0.420
# 3  3.00  0.131

Sometimes I'll wrap hive-named functions (like percentile) around R functions (quantile in this example) so that my stuff works in both spark and local dplyr mode.

Hi Matt and Harry,
Thanks for the replies.  I realized today what I was really looking for was hash, which takes an input and creates a number.
Best,
Nathan

I'll watch out for percentile, as that would be useful.  Currently I use ft_quantile_discretizer, and the hive function would be much more convenient.


Invalid SodaResult Values

I don't know if this is an issue with SODA.Net or with Socrata, but we've been getting incorrect SodaResult responses when we perform batch upserts.  Updates to one of our datasets is returning only RowsUpdated, even when the upsert creates rows.  Another batch update for a similar dataset returns 0 for all three rowcount results.  Both updates return valid batch counts and no messages or errors.
Is anyone else getting unusual SodaResults like this for batch upserts?

@atguser it's hard to say without knowing more about your exact dataset/situation. SODA.NET just parses the response coming back from Socrata directly. It is possible that for Upsert requests, Socrata reports all actions as "Updates".
You can check out the raw response from Socrata by debugging into SodaRequest.ParseResponse.

Thanks. I'll add that to my debugging toolbox!
Looks like the "no count" results was something in the Socrata side.  I've still got a question in to them about whether batch upserts only return RowsUpdated or whether they also return RowsCreated.  I'm currently getting only RowsUpdated results, even when I have new rows in the batch.

Closing for now. Please feel free to open a new issue if a new problem/question arises.


Using scrape in artoo.spiders

Hello @suntong001,
Does this example help you?

Thanks for the reply. I saw
artoo.ajaxSpider(
  function(i, $data) {
    return nextUrl(!i ? artoo.$(document) : $data);
  },
  {
    limit: 2,
    scrape: scraper,

I..e, I saw a scraper there, but I can't figure out how the second-step scraping is specified and how it works.

artoo.ajaxSpider(

  // This function is an iterator.
  // Its aim is to return the next url to fecth or false if you want to stop
  //-- 'i' is the index in the iteration of urls
  //-- '$data' is the jQuery-parsed data of the last fetched url
  function(i, $data) {

    // nextUrl is a function that take a jQuery selector and returns
    // the next url to fetch

    // If !i then, we are only starting the spider meaning that the next url
    // is available on the current page rather than the last fetched one.
    return nextUrl(!i ? artoo.$(document) : $data);
  },

  // Spider's settings
  {

    // We want to fetch a maximum of two pages
    limit: 2,

    // We are going to scrape the pages using the scrape definition written above in the doc example
    scrape: scraper,

    // We want to concatenate results so we have [title1, title2, title3, title4]
    // rather than [[title1, title2], [title3, title4]]
    concat: true,

    // Final callback fired when the spider retrieved everything
    //-- 'data' is the scraped data
    done: function(data) {
      artoo.log.debug('Finished retrieving data. Downloading...');
      artoo.savePrettyJson(
        frontpage.concat(data),
        {filename: 'hacker_news.json'}
      );
    }
  }
);
 ```

Oh, thank you, thank you.
I was just about to say, that it is helping, having been staring it long enough. Your comment make it very clear now. Thank!

The only thing left not so clear to me is how the data flow -- I was trying to guess the relationship between the keyword data in var scraper and in done: function(data), and frontpage.concat(data). Also I'm not sure where the $data comes from in function(i, $data). My guess that none of them are related, but brings up a more important question, how the data are actually passed around? Thanks.

The data keyword in var scraper works like a named argument because this is what the scrape methods expects when you use it with only one argument.
In done: function(data), data is the aggregation of the spider's retrieved data.
$data in the iterator function is the html of the last page retrieved by the spider and parsed by jQuery (this is why $ is used).

Got you. Please close. Thanks.
BTW, wiki updated.

Hi again Yomguithereal,
I tried the artoo.ajaxSpider for the first time, but I only get the first page result. I.e., the spider is not working. Can you see what I'm doing wrong please? Here is the code:
var dmList = artoo.scrape('tr td a[href*="edit_domain_id"]', {
  domain: 'text'
});

artoo.ajaxSpider(
  function(i) {
return 'http://freedns.afraid.org/domain/registry/?sort=2&q=.us&page=' + (i+1);
  },
  {
scrape: dmList,
limit: 9,
concat: true,
throttle: 500,
done: function(data) {
  artoo.log.debug('Finished retrieving data. Downloading...');
  artoo.saveCsv(dmList);
}
  }
);

I.e., if you visit http://freedns.afraid.org/domain/registry/?sort=2&q=.us&page=1, then you will see that there will be 9 pages altogether, but I only got one. What's wrong?
Thanks

The option you are passing to the scrape property is not valid @suntong. What you should really be doing is the following.

var definition = {
  iterator: 'tr td a[href*="edit_domain_id"]',
  data: {domain: 'text'}
};

var initialList = artoo.scrape(definition);

artoo.ajaxSpider(
  function(i) {
    return 'http://freedns.afraid.org/domain/registry/?sort=2&q=.us&page=' + (i+1);
  },
  {
scrape: definition,
limit: 9,
concat: true,
throttle: 500,
done: function(data) {
artoo.log.debug('Finished retrieving data. Downloading...');
artoo.saveCsv(initialList.concat(data));
}
  }
);

Oh, thanks a lot, I had always been wondering how the data flow. Now everything is clear. Thanks.


Added digest to hash-password

On node.js version 6 the digest on crypto.pbkdf2 is required.
Default is 'sha1‘.

This fix is required for Node.js 8. In Node.js 6 and 7 the following warning was printed:
(node:35) DeprecationWarning: crypto.pbkdf2 without specifying a digest is deprecated. Please specify a digest
In Node.js 8 the function has been removed and results in an unhandled exception in Hoodie.
Changing this file manually in node_modules fixes the issue. I believe the only thing that is required is to release a new version of this module on npm with this change that has already been merged.


Add Faculdade Sapiens

@leoalmeidasa First stage of verification passed, official domain is faculdadesapiens.edu.br as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.


enable upload same consignment on different blinded UTXOs

This PR removes a check that prevented the upload of the same consignment file on different blinded UTXOs.
The new check unlocks that possibility while also preventing to change of a consignment file for the same blinded UTXO and making the API idempotent in case of re-upload of the same file on the same blinded UTXO.

Great! thanks 😄


feat: allow usage of enabled flag for resource creation

Description

Motivation and Context

Breaking Changes

How Has This Been Tested?

[ ] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[ ] I have tested and validated these changes using one or more of the provided examples/* projects

[ ] I have executed pre-commit run -a on my pull request

This PR is included in version 1.3.0 :tada:


UI thread becomes unresponsive while loading commits

Description:
The app suffers from a performance issue where the page becomes unresponsive while loading more commits which seems to be exacerbated by poor internet and/or git repositories that have lots of branches.

Possible fix:
Lengthy or blocking javascript code should be run on a web worker to prevent from blocking the UI. As I've looked through the code I noticed that async functions are used frequently, however these will still block the UI thread. Web workers are fairly limited in what they can do (for instance they can't modify the DOM) so the fix can be anywhere from simply rewriting a few async functions to use web workers to requiring a code rewrite with workers in mind. I'll need to look at the code some more to give a more definitive answer.
Version information:

Windows 10
Chrome 109.0.5414.120 (64 bit)
Le Git Graph 1.1.2

Seems like this is the most serious issue presently. 3-4 clicks on load more and the extension practically becomes unusable. I have tried having a look at the code to see for possibile fixes, and I will look into it further.
It would be really helpful if you could share details about why this happens if you come across some insights.
@BitBlitObviMormon

With that last paragraph in mind this is probably also related to issue #3.

The performance issue caused due to higher number of branches has been listed as a new issue, since that would require a completely different way of approaching. That issue is due to the lack of required APIs from GitHub, and would require some hacks/workarounds. It can be tracked at #41


Redis cache always miss?

I'm running a simple webserver and trying to setup redis based cache for it -- as in-memory one would fail in prod use-cases.
This is the attachment of the middleware
// using redis to cache HTTP requests
	redisStoage := fiber_redis.New(fiber_redis.Config{
		Host: "localhost",
		Port: 6379,
	})

	app.Use(cache.New(cache.Config{
		Expiration: 15 * time.Second,
		Storage:    redisStoage,
		Methods:    []string{fiber.MethodGet, fiber.MethodHead, fiber.MethodPost},
	}))

has the cache always miss

I was initially trying to use the connection URL based method
// using redis to cache HTTP requests
	redisStoage := fiber_redis.New(fiber_redis.Config{
		URL: config.REDIS_CLUSTER_URL,
		Reset:    false,
	})
	app.Use(cache.New(cache.Config{
		Expiration: 15 * time.Second,
		Storage:    redisStoage,
		Methods:    []string{fiber.MethodGet, fiber.MethodHead, fiber.MethodPost},
	}))

I am confident that

redis is working -- as other caching methods are working
the connection string is working -- as app cache layer uses the same

@YashKumarVerma Have you check if the entries are getting created inside redis? Try using redis-cli to verify.

yes I did manually check via redis-cli, there are no entries in it

finding:

created a simple cached app (https://github.com/YashKumarVerma/fiber_redis_conn)
ran locally - cache working
ran on server - cache not working

restarting redis container fixed the issue

checked logs
turns out some issue with redis container

1:C 07 Oct 2022 04:39:13.989 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 07 Oct 2022 04:39:13.989 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 07 Oct 2022 04:39:13.989 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 07 Oct 2022 04:39:13.989 * monotonic clock: POSIX clock_gettime
1:M 07 Oct 2022 04:39:13.989 * Running mode=standalone, port=6379.
1:M 07 Oct 2022 04:39:13.990 # Server initialized
1:M 07 Oct 2022 04:39:13.990 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory
 = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 07 Oct 2022 04:39:13.990 * Ready to accept connections
1:S 07 Oct 2022 09:40:51.561 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new
master with just a partial transfer.
1:S 07 Oct 2022 09:40:51.561 * Connecting to MASTER <IP_ADDRESS>:8886
1:S 07 Oct 2022 09:40:51.561 * MASTER <-> REPLICA sync started
1:S 07 Oct 2022 09:40:51.561 * REPLICAOF <IP_ADDRESS>:8886 enabled (user request from 'id=18 addr=<IP_ADDRESS>:32776 laddr=<IP_ADDRESS>:6379 fd=8 name= age=0 idle=0 f
lags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=47 qbuf-free=20427 argv-mem=24 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22320 events=r cmd=slaveof user=def
ault redir=-1 resp=2')
1:S 07 Oct 2022 09:40:51.723 * Non blocking connect for SYNC fired the event.
1:S 07 Oct 2022 09:40:51.885 * Master replied to PING, replication can continue...
1:S 07 Oct 2022 09:40:52.209 * Trying a partial resynchronization (request 3b1da34756aa9b851bb5a94958bda7cc0a458fd3:1).
1:S 07 Oct 2022 09:40:52.374 * Full resync from master: ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ:1
1:S 07 Oct 2022 09:40:52.375 * MASTER <-> REPLICA sync: receiving 54976 bytes from master to disk
1:S 07 Oct 2022 09:40:52.698 * Discarding previously cached master state.
1:S 07 Oct 2022 09:40:52.698 * MASTER <-> REPLICA sync: Flushing old data
1:S 07 Oct 2022 09:40:52.698 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 07 Oct 2022 09:40:52.704 # Wrong signature trying to load DB from file
1:S 07 Oct 2022 09:40:52.704 # Failed trying to load the MASTER synchronization DB from disk: Invalid argument
1:S 07 Oct 2022 09:40:52.705 * Reconnecting to MASTER <IP_ADDRESS>:8886 after failure
1:S 07 Oct 2022 09:40:52.705 * MASTER <-> REPLICA sync started
1:S 07 Oct 2022 09:40:52.866 * Non blocking connect for SYNC fired the event.
1:S 07 Oct 2022 09:40:53.027 # Failed to read response from the server: Connection reset by peer
1:S 07 Oct 2022 09:40:53.027 # Master did not respond to command during SYNC handshake
1:S 07 Oct 2022 09:40:53.900 * Connecting to MASTER <IP_ADDRESS>:8886
1:S 07 Oct 2022 09:40:53.901 * MASTER <-> REPLICA sync started
1:S 07 Oct 2022 09:40:54.064 * Non blocking connect for SYNC fired the event.
1:S 07 Oct 2022 09:40:54.227 # Failed to read response from the server: Connection reset by peer
1:S 07 Oct 2022 09:40:54.227 # Master did not respond to command during SYNC handshake

Marking as resolved as was not related to fiber -- but underlying infra - redis.

Marking as resolved as was not related to fiber -- but underlying infra - redis.

Glad you finf your issue!


add --launch-container option in launch command

This PR will add --launch-container option in launch command

ok, let's sync with @tcrespog for the sdk

PR to add in sdk
https://github.com/seqeralabs/tower-sdk-gencode/pull/8

PR to add in sdk seqeralabs/tower-sdk-gencode#8

this SDK PR is not correct, checking with @JaimeSeqLabs to fix it

SDK pr has been fixed

@tcrespog is there recent chnages in GoogleBatchPlatform?
Tests are failing for it with new tower sdk

org.opentest4j.AssertionFailedError: expected: <> but was: <
 ERROR: The program tried to reflectively invoke method public java.lang.String io.seqera.tower.model.GoogleBatchConfig.getNextflowConfig() without it being registered for runtime reflection. Add it to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.

 ERROR: Connection error. Check the connection using the command 'tw info'.
>
	at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
	at app//org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
	at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
	at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
	at app//io.seqera.tower.cli.BaseCmdTest.assertOutput(BaseCmdTest.java:183)
	at app//io.seqera.tower.cli.computeenvs.platforms.GoogleBatchPlatformTest.testAdd(GoogleBatchPlatformTest.java:61)
	at java.base@17.0.8/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base@17.0.8/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base@17.0.8/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

is there recent chnages in GoogleBatchPlatform?

Yes, in In GoogleBatchConfig and others: https://github.com/seqeralabs/platform/pull/6691.

Tests are failing for it with new tower sdk

The program tried to reflectively invoke method public java.lang.String io.seqera.tower.model.GoogleBatchConfig.getNextflowConfig() without it being registered for runtime reflection

I'm assuming the tests failing here are the native ones.
This is most likely because we need to manually add the new method signature to the reflect-config.json file.

Was the --launch-container tested?

Was the --launch-container option tested?

I am not sure, how to test it?

Run a local instance of Seqera Platform
Point tw-cli to the local instance: export TOWER_API_ENDPOINT=http://localhost:8000/api
Do a launch using the new parameter: tw --insecure ... --launch-container ...

This option allows you to change the nextflow launcher used by Platform when running a workflow. Use for example quay.io/seqeralabs/nf-launcher:j17-24.07.0-edge.
You may want to use https://cloud.stage-seqera.io/ for sake of simplicity and run this "workflow" https://github.com/pditommaso/nf-sleep

Ok finally my pipeline runs and i can test it successfully
client:

Excellent.


Fixes #78 : using uppercase for 'Application' service

Fixes #78

It need to use uppercase when we require zend-servicemanager:v3

@samsonasik That's not really a fix. We need for both application and Application to resolve to ensure user-land code continues to work. This will ensure the internals work, but it's the user-land code I'm more worried about.

@weierophinney I've added new entry for 'application' in the ServiceListenerFactory as it won't work with alias due Circular alias reference

should be ok now?


Investigate IVSHMEM or VirtIO support

As discussed on #35 and on Reddit I'm studying a way to have high quality, high performance audio from windows VMs to linux host with IVSHMEM or VirtIO.
The idea is to have a registry key to swith from the default networked mode to a SHM mode.
In practice I'll need to create something that mimic savedata.cpp
I have to find the documentation for IVSHMEM or VirtIO and see how much there is to do.
The receiver should be a piece of cacke. Read from an hypotetical /dev/scream-shm , write to pulse, done.
I'll start working on this after #35 is finished and merged upstream.

You bet I'll test (and ultimately use) this too!

Good! Because I have something for you :)
Before you get all excited, no, I have not finish it yet, but I have started working on it and I have a small benchmark to test on as much machines as possible.
HOWTO INSTALL

Add a new IVSHMEM device like this. Size should be 1M.

    <shmem name='looking-glass'>
      <model type='ivshmem-plain'/>
      <size unit='M'>64</size>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
    </shmem>
    <shmem name='scream-ivshmem'>
      <model type='ivshmem-plain'/>
      <size unit='M'>1</size>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/>
    </shmem>

It is REALLY important that the new device is enumerated after the one used by looking glass.
Currently there isn't a mechanism to differentiate between a device and another.
While I'll make sure to introduce something looking glass doesn't bother to check. It use the 1st available shared memory.
If you swap the order you'll send audio to looking glass and video to scream. It may be a nice psychedelic show, but it's best to avoid it.

Start your VM.
You should see in /dev/shm/ that a new device named scream-ivshmem was created.
Ensure you have read permission.

In windows ensure you see 2 IVSHMEM devices in device manager, like this.

Download the test profram from here

Run the benchmark, let it run for at least 10 seconds, then post a screenshot like this

The reason for this test is to ensure that the transfer speed is high and consistent among different KVM configurations. That's why I need as much tests as possible in different environment.

Not needed, but if you are curious you can hexdump /dev/shm/scream-ivshmem. You should see a 16 byte header follower by a debug string. If you keep looking you should find a sequence of blocks of 1280 bytes in size filled with 0x01 in the 1st block, 0x02 in the 2nd and so on.
Those blocks are the chunks that will contain the scream packets. They are memory aligned for faster memory copy.

Roadmap
The scream-ivshmem-pulse receiver is almost done. It's the easyest part of the project.
What I'll do next is to modify the test program I posted here to send audible chunks instead of garbage, to test the system.
For last, when the code is stable, I'll move it from the test program to the scream driver. And that's it I think.
I'll take the time to patch all the linux receivers of course.
Unfortunately I will not have much time this week so don't expect anything ready soon, but maybe I'll send other test programs.

Was about to run your test program but it seems like it's a debug build?
This would require me to install visual studio to get some missing dlls.
Mind doing a release build for it? Would save me a lot of trouble 😅

Oops, you are right, my bad. I'm at work now, I'll build a release version this evening.
In the meantime I've implemented a lot more, I'll give you access to scream-ivshmem-pulse that is already working.
The build version I'll send will require you to pass a raw pcm file as an argument. At the moment it's only 44100Hz, 16bit, 2 channels, but only for my convenience. Full multichannel support is only few lines ahead.
To make a raw pcm file you can take any audio (or video) file as input and use ffmpeg like this:
ffmpeg -i <input-file> -f s16le -acodec pcm_s16le test.raw
Then in windows you can run
scream-ivshmem-text.exe test.raw
And in linux
scream-ivshmem-pulse
You should hear your test sample playing flawlessy. As a test you can kill and restart the windows binary as many time as you want while keeping the linux one running. If everything is ok it should restart playing the audio sample without crashing.
Same with the linux receiver, you should be able to kill and restart it at any moment without having to restart the windows binary. At least that's the idea.
If it crash or it produce strange behaviours let me know.

Hi @tadly , you can download both scream-ivshmem-text.exe in release mode and scream-ivshmem-pulse source code here
Follow the instructions on my previous message. If you encounter any problem let me know.
While you are listening to some audio can you check windows task manager to measure the process usage? CPU usage should be near 0%, while memory is dependent on the file you are streaming (it's loaded in ram at the start).
Please send me the output of both windows and linux program, thank you.

Hi everyone, figured it might not hurt posting my findings. The demo works great, I have less than 2% CPU usage, most of the time its about 0.5%.
I am a bit surprised about the rather low TX speed after the initialization, but had no dropouts. Using QEMU v4.0.0-rc3, without libvirt.
Outstanding work @martinellimarco! I will definitely use this when its ready!

Hi @alegru , thank you for testing this!
If you use the first exe, the benchmark, you'll see unconstrained speed (100+Mbps in most cases) since it's sending random data without waiting for the receiver to pick them up.
The second one is a fully functional transmitter. It will fill the audio buffer as fast as possible (the initial spike you see). Then it will wait for the receiver to free some space in the buffer before writing anything else.
The speed you see is a constant. 44100Hz, 16bit, 2 channels = 44100162/1024/1024 = 1,34 Mbps.
It will not be any faster, it doesn't have to. Instead the thread will sleep for a bit, keeping the CPU utilization low.
The fastest it will have to be is at 192kHz, 32bit, 8 channels. That's around 50Mbps, but it can handle it.

Thank you for explaining the details! Throttling on purpose makes sense, guess I was a bit distracted because of your first screenshot with unthrottled random data.
If you need something else to test I will gladly post feedback.

I will probably make another release of the transmitter with more command line parameters to specify the sample rate, sample size, number of channels and speaker configuration, to test the performance of the system with any kind of configuration and the stability switching from a configuration to another.
It's not too much work, really a few lines of code, but I don't have much time this week, I can work on this only at night. I expect to release this update this evening or tomorrow morning (Italian time).
This will require different raw pcm files, with matching parameters for each test. I'll take the time to document how to make those files with ffmpeg.

Was trying to get it tested but am struggling a bit.
I'm getting Unable to enumerate the device, is it attached? 259 but my configuration seems to be fine from what I can tell.
I even removed the looking-glass shmem (including libvirtd restarts) to make sure there are no conflicts etc.
<shmem name='scream-ivshmem'>
    <model type='ivshmem-plain'/>
    <size unit='M'>1</size>
    <address type='pci' domain='0x0000' bus='0x0b' slot='0x01' function='0x0'/>
</shmem>

-rw-rw---- 1 tadly kvm 1048576 Apr 16 10:48 /dev/shm/scream-ivshmem

Don't ask me why but now it's working (Re-added looking-glass, did some more reboots)
Anyways, here be your stats fine sir :)
CPU initially climbed by ~1% for a few seconds (could be something else) and settled back to 7% after.

7% ? it's a lot :( I was expecting it to stay way lower, I'll have to work more on this.
Anyway, glad it's working. There is a second parameter I've not mentioned right after the file name, it's the device index. It default to 1 but you can change it. The assumption is that the 0 is looking glass and the next one is scream, but if you remove looking glass you can pass 0 and it should work.
I'm studying a robust way to scan IVSHMEM devices and detect what is the one assigned to scream, but it's not that easy.
As a fallback I'm thinking of having a registry key to specify the device path that windows see. This way the user will always be able to override the autodetected device. I have not done any work in this regard yet.
Anyway, thanks for the test, it's good to see positive feedbacks :)

No no no you misunderstood :)
7% is my VM at idle (for whatever reason) and running your tool does not increase this, so all is good ;)
I assume the windows path is quite an ugly one (unlike it is on linux).
I do think we should talk to the guys over at looking-glass as well for them to not assume the device.
Given how devices can't be named or anything, specifying an index is probably the nicest solution while both projects should simply default to 0.
Sure, logic could be added to e.g. check its size and determine if it's sensible to use it but that's still kind of hacky.
What I meanis, looking-glass could say: that thing's only 1M big, I can't use that
while you could say: that things >=32M, that's probably not right
Personally I've always favored explicitly doing stuff and as we're working on something that's not for the dumb consumer it doesn't have to be plug&play anyways so device-index or device-path (if documented) is IMO just fine :)

I just noticed on the host, the CPU usage of pulseaudio and scream-ivshmem-pulse is close to 0% while the gust is transmitting audio, but 100% for scream when the transmit has not started / is finished. If the host scream process is not started, the guest uses about 20% CPU while waiting for the host to initialize the shared memory. Might need some synchronization mechanism in the driver later.

Thanks for reporting this. The reason is that at the moment I'm doing "busy waiting" for the transmitter to send. This will absolutely change in the final version and you'll always see cpu utilization at a minimum.

Hi, @martinellimarco I'm yestaes from reddit. I'm glad to see that you are working on this project.
I came here to show my results as the other here.
https://i.imgur.com/VYfGa2z.png
https://i.imgur.com/ogIRxs2.png
I noticed the same issue as well as alegru told you.
Regards

Hi guys, another day, another release
@1985a Hi yestaes, it's nice to see you here. Thank you for testing this, I really appreciate it.
In the new release there is:

a fix for the high cpu usage of the client while waiting for the transmitter.
improved transmitter cpu utilization
improved transmitter memory usage
full support for all the sample rates, sample sizes, number of channels and channel mapping configurations :)

At 192kHz, 32bit, 8 channels this is the kind of performance you should see for the transmitter. CPU ~1%, RAM < 1MB, average speed ~50Mbps. And, best of all, no crackling at all. :)

To create a test file for this configuration run ffmpeg like this
ffmpeg -i <input-file> -ac 8 -ar 192000 -f s32le -acodec pcm_s32le test.raw
The source can be any audio or video file, even if the number of channel, sample rate or sample size are lower. Beware that even few minutes of audio will produce a file of a few GiB.
The file I was testing for the screenshot above was 1.5GiB
To test it you'll have to pass the appropriate parameters to the transmitter.
scream-ivshmem-test.exe test.raw 192000 32 8 255
Where 192000 is the sample rate, 32 is the sample size, 8 is the number of channels and 255 is the channel map.
Now, if you have any problem with channels mapped in the wrong position you can replace 255 with 3. This will default to stereo and pulseaudio will mix the 8 channels to 2. It's not ideal but it works.
Not needed, but maybe you want to play around
If you want to map the channels differently you can calculate the right number for your raw pcm file summing together the numbers from this table.
| FRONT_LEFT | 1 |
| FRONT_RIGHT | 2 |
| FRONT_CENTER | 4 |
| LOW_FREQUENCY | 8 |
| BACK_LEFT | 16 |
| BACK_RIGHT | 32 |
| FRONT_LEFT_OF_CENTER | 64 |
| FRONT_RIGHT_OF_CENTER | 128 |
| BACK_CENTER | 256 |
| SIDE_LEFT | 512 |
| SIDE_RIGHT | 1024 |
In the example it's assumed that in the pcm file the channels are from 1 to 8 in this exact order. Summing the numbers from the 1st to the 8th you get 255.
If for any reason the channels in the raw pcm appear in an order that is not as above (eg, LOW_FREQUENCY before FRONT_CENTER) then you can't make a correct channel map.
This is not a limitation for the final product, windows will always output its stream in that order.
NOTE
There is a known bug if you use an odd number of channels (eg 6.1 surround). Don't test them now, the receiver will just crash.
This bug also affect the 3.0 release of scream, I'll patch it soon.
Enjoy! :)

Hey everyone, time for feedback. As far as I can test, everything works as expected, but since I have a stereo only setup someone else should probably comment on the multi-channel side of things. So far, CPU utilization is much lower, always close to 0%. Only pulseaudio was working hard downsampling the 7.1 stream, which was expected. There were no audio dropouts noticable.
Like in my last comment, if the scream-ivshmem-pulse is not running and the transmitter already started, we have a CPU usage of about 20% on windows. This is actually not really problematic, since best practice would suggest the host process should already be running, waiting for the windows driver to start, not the other way around. But I am sure this is easy to fix.
Is it normal for the ivshmem address to change? I did not reboot between the two tests; is the windows process mapping the ivshmem in its process space? Will certainly study the source code when you publish it :)
As always, thank your very much for your hard work. I am sure this will supersede other virtualized audio solutions.

Hi @alegru , thank you for your test.
Yes, it's normal that the IVSHMEM address change, for the exact reason you guessed.
It's good to know that there is no crackling on your system as well.
The high CPU utilization you see now if there is not an active receiver will completely disappear in the final product.
The thing is that at the moment the transmitter keep waiting in a loop for the other process to pickup. Once this will be converted into a proper audio driver windows will notify it once there is a "packet" to send and the driver will do absolutely nothing for the rest of the time.
If there will not be a receiver on the other side the audio will be lost, just like it happen now with the network version.
What matters is the cpu / memory usage during the actual transmission, that's the only part of the code that will end up as is in the final product.

@martinellimarco Hi, I'm glad to see you that you are making progress in this project.
I'd like to test this, but today I have to work, so tomorrow I'm going to test this progress.

I tried to run the test program on a Windows 7 Guest but it did not run.

Hi there. I don't know what is the problem here today
https://imgur.com/a/DbqkT53
My user has read permission to the dev file. On the other hand, the receiver is waiting for scream server and this just it does not start.
Regards

@1985a that's definitively the issue, the transmitter is picking up looking glass.
Can you post here the XML part where you define the 2 IVSHMEM devices ? I need to see both looking glass and scream.
Have you changed something in the VM or was it working then after a reboot it stopped ?
Anyway, there is another parameter you can add after the last one, it's the device index. By default it takes 1, it's assumed that looking glass is the 0.
If for some reason you have them swapped you can try to add a 0 as last parameter.
@Vaporeon Thank you for taking the time to test this. Can I ask you to add more details please? What was not working exactly?
If you can please post:

XML of your VM where you have defined the IVSHMEM device
a screenshot of windows device manager where you can see that device
the output of ls -la /dev/shm/
a screenshot of the receiver on linux
a screenshot of the transmitter on windows
any other info that you think may help

@martinellimarco hi there.
Here is the part of the IVSHMEM devices.
 <shmem name='looking-glass'>       <model type='ivshmem-plain'/>       <size unit='M'>64</size>       <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>     </shmem>     <shmem name='scream-ivshmem'>       <model type='ivshmem-plain'/>       <size unit='M'>1</size>       <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>     </shmem>

@1985a Thank you. I was hoping for a mistake in the XML, instead it's what I was fearing.
You see, in your XML you have set two devices on the same bus, one in slot 2, one in slot 3. That's perfect.
So far in every test I did Windows respected that order. I didn't know if that was always the case, I have not looked into it in deepth.
I was hoping the order in windows was always predictable.
The user would have had to set the device index once and then enjoy his new setup forever.
In practice your report proved that it's not predictable and that windows can reorder the devices even if the user doesn't touch the configurations.
I'm curious. Does looking glass still work even if the devices are swapped? I was under the impresion that looking glass always used the first IVSHMEM device available, but maybe it skips the 1MiB one because it's too small.. I'll have to take another look at their source code. This is critical because I don't want this to interfere with looking glass in any way.
Anyway, I have a couple of ideas for autoselecting the right IVSHMEM device. If looking glass can't do this for any reason maybe I can contribute a patch to their code too.

@martinellimarco Haven't had time to test your new stuff yet (I'm really sorry) but can at least answer your question in regards to looking glass...
Looking glass will just fail if the device it picks is to small. Don't remember the exact issue but for me, it never worked with both ivshmem configured.
This is also why I previously mentioned that we should open a issue over at looking-glass so they can harden their implementation as well. I'm referring to my comment here
Browsing through the the registry it at least seems like windows isn't recreating them with new ids.
So we could either:

require the user to specify the Device Path/Hardware ID
More work for the user but less likely to cause issues
Check how big the memory device is and decide whether or not it's feasible to use.
Less/no work for the user but more likely to go south if other memory devices exist

looking-glass would have to do the same.
For reference:

@martinellimarco On windows 7 Scream-IVSHMEM-test.exe Crashes on run with the arguments test.raw 192000 32 8 255 0, using a ~1.6GB test file.
I have only one IVSHMEM Device and am not using looking glass, I also only run Qemu from the command line.
The relevant arguments are:
-device ivshmem-plain,memdev=ivshmem
-object memory-backend-file,id=ivshmem,share=on,mem-path=/dev/shm/scream,size=1M

The host side pulse application reports it is waiting for scream to initialize the shared memory.
Also, I'm using the IVSHMEM device driver found here:
https://github.com/virtio-win/kvm-guest-drivers-windows/issues/217

Regarding device order, I just tested scream and looking together. It works fine if specified like this on the command line:
OPTS+=" -device ivshmem-plain,memdev=ivshmem-looking-glass"
OPTS+=" -object memory-backend-file,size=32M,id=ivshmem-looking-glass,share=on,mem-path=/dev/shm/looking-glass"

OPTS+=" -device ivshmem-plain,memdev=ivshmem-scream"
OPTS+=" -object memory-backend-file,size=1M,id=ivshmem-scream,share=on,mem-path=/dev/shm/scream"

As soon as I switched the order, Windows also enumerated them with scream first and looking glass second. Of course, since looking glass is hard coded to use the first device, it crashes silently; but scream works if I specify device 0 on the command line.
Maybe I am lucky, but on my setup Windows orders the devices just as specified on the qemu command line...

@alegru and @tadly , thank you for confirming that looking glass fail if the devies are in the wrong order. I will get in contact with them to find a solution.
@alegru In all my tests windows always respected the order of the devices and thanks to that looking glass never failed, but the report of @1985a showed that this is not always the case, so yes, I think both of us are lucky.
I remember a similar situation in linux where devices (usually hard disks or network cards) showed up in different orders depending on the time the controller took to initialize the device. Maybe this is something similar, while most of the time the devices are initialized in the same order sometime they end up reversed. I'm really speculating here, I'll have to dig a bit deeper in the details.
@tadly the hardware IDs are the same for both IVSHMEM devices. There are other differences like the "Device BIOS Name" or the "Physical Device Name", but I don't know if when the device are detected in a different order those IDs change. Sorry, I don't have the names in english, I'm translating them and the actual names may be differents.
I can think of a few solutions (ignoring for a moment looking glass)
Option 1
as @tadly said, the driver will check for the memory size and will select the device with less memory.
Pros: very easy
Cons: not future proof. Now we are dealing with only 2 devices, but what if some day someone else invent a new something that requires IVSHMEM?
Option 2

the receiver write a magic number at the beginning of the shared memory
the transmitter / driver loop trought the IVSHMEM devices and it peeks the first bytes in search of the magic number
if no such number is found then the transmitter will retry at a later point in time, otherwhise from that point onward that device will be the chosen one
Pros: easy, future proof, it doesn't require any configurationa from the user
Cons: I don't know if this can interfere with looking glass. I have to read their source code in details.

Option 3

investigate if IVSHMEM IOCTLs provide a way to identify the devices

@Vaporeon thank you, I'll make sure to test on win7, but I don't have one at hand now, it will require some days. In the meanime can you test with stereo? (44100 16 8 3 0)

@Vaporeon  is it possible that IVSHMEM is not working in Win 7 ? See this

Good morning everyone here.
@martinellimarco you have to excuse me. Today while I was brushing my teeth it came to my brain an image. That day I was making a tutorial for my personal blog, so I installed a new USB device, a network card then uninstalled it.
BTW, this is my blog. It is in spanish, but I drop the things I consider very important to save.

@1985a oh! Ok, so maybe that's the reason.. I'm investigating what's the supposed behaviour. At the moment I'm setting up different VMs with different configurations and multiple IVSHMEM devices each to test the edge cases. We'll see.

Just saw (and tried) this and its awesome :+1:
Have you already tried integrating it in scream? Or can you share the windows application? I'd like to use it as my audio device for my (new) looking glass system :)

I know I'll make myself unpopular by saying this but it really grinds my gear when people don't even spend the necessary 10 seconds to read the original post to see what this issues intent is.
Obviously @martinellimarco is trying to integrate it into scream (hence why this issue exists) and if he would have working prototype he would have posted so already.
I'm not trying to be a dick (even though I sound like one right now) but please, just spend the few minutes to go through the issue (in general, not just this one) FIRST before posting irrelevant stuff notifying everyone involved etc.
Sorry for the (also kinda irrelevant) rant.

@tadly sorry for my previous post. I didn't managed to say what I meant with it.
I've read most of this thread and really like it. What i wanted to know is if @martinellimarco already started at the implementation in scream or if he would share his sample application (i don't know it this application uses scream or code from it in some way) so it would be possible to start implementing it in scream while he works on the device detection. I do programming in C/C++ myself and would like to help :)

@paul70078 I'll share it when it's ready to test. At the moment you'll get a BSOD so... not yet.
I was hoping to release something this weekend, but there is a bug I'm facing with the kernel memory manager, due to my lack of experience in windows driver development I guess. I know I will fix it, but I don't know how long it will take. If you have experience with this any help is well accepted.
Aside from this, here's a checklist of what I've done and what's left to do.

Task
Status

Userspace tests to see if this is viable
Done, it's the program I've already published here

Have Scream detect a registry key to enable IVSHMEM instead of network
Done

Have Scream switch from network to IVSHMEM
Done, original Scream have CSaveData, the new one have CSaveData and CIVSHMEMSaveData

Enumerate IVSHMEM devices from kernel space
Done, I can enumerate all the IVSHMEM devices (so far tested with 10 devices)

Do IOCTLs to enumerated devices
Done, I can talk to IVSHMEM devices from kernel space! :D

Detect the right IVSHMEM device based on device size and order
Done, I can detect the size of all IVSHMEM devices. At the moment the rule is "select the 1st one that have size = 1MB". The size will be configurable.

Have access to the shared memory (PCI BAR2 of IVSHMEM)
Done, I can read and write, linux can see this in /dev/shm/

Pass configurations such sample rate, sample size, channels, channel mapping and status from Scream to the receiver
(Almost) done. The configurations are passed correctly, the receiver detect them and setup the correct Pulseaudio parameters, but I think I'll have to review a few things here.

Write the PCM samples from windows internal buffer to the shared memory.
BSOD :( I'm studying how to fix this at the moment, but it takes ages. Windos crash, reboot a few times, go in recovery mode, uninstall the driver, reboot, try another one, BSOD and repeat. I'm debugging with WinDBG, if someone have experience with it any tip to speedup this process is appreciated.

Performance check
WIP

Test on multiple setups and environments
ToDo, here i'll ask your help guys

Review it all and clean the code
ToDo

Request a PR
ToDo

Regarding the BSOD, I can take a look at the code if you like. Is it checked into your clone repo?

Not yet, the code is a bit messy, I want to clean it up first.
I think I have just figured out the root cause, I'll try to explain so maybe you can tell me if I'm on the right track, or if you know how to deal with it.
At the moment I do an IOCTL to IVSHMEM to request a pointer to the shared memory.
This work, I can read and write from the function that called this and other functions called in the same context.
I then save this pointer for later use. When WriteData is invoked (suppose we are talking about savedata.cpp, I strarted from there) I try to read / write from that same pointer and it crash (but not 100% of the time).
Looking at IVSHMEM source code here I see that they call MmMapLockedPagesSpecifyCache with the UserMode so the address that they return to me is not always valid.
Do you know if there is a way to "convert" this to KernelMode so I access to it in any context?
I'm reading about MDLs but I admin that's a concept I have yet to grasp.
I think a solution would be to send a PR for IVSHMEM to have a parameter in the IOCTL to select UserMode or KernelMode but I fear this will take ages to be merged and distributed.

I think you should not use pointers to foreign driver mem in the interrupt hot path. Did you try keeping the internal ring buffer, and copying the data further in a worker (like the networked version does with CSaveData::SendData?

Yeah, I totally agree with you and that's how I'm doing it,  but it's the same thing, the worker doesn't have access to that memory address too.
The code is organized the same way as CSaveData, with WriteData that fill a ring buffer and launch a worker that when invoked execute SendData.
In SendData it crashed 100% of the time, so to better understand the problem I've started testing where it was working and where it wasn't, going backward (SendData -> SendDataWorkerCallback -> WriteData).
Finally in WriteData I had some success, it crashed most of the times, but once in a while I could see the audio chunks being copied in the shm from linux, that's why I mentioned it.

I see. What bugcheck code do you get when writing in SendData?

Bugcheck 7E, exception code 0xc0000005, attempt to read from address 0x000001bede7f004.
That's the address returned by the IVSHMEM IOCTL during initialization.
As a test, I'm thinking of calling again the REQUEST_MMAP and RELEASE_MMAP IOCTLs in SendData to see if it works. I have to go out an hour, then I'll test this if you don't have another idea.

OK then I guess your assumption here is correct:
Looking at IVSHMEM source code here I see that they call MmMapLockedPagesSpecifyCache with the UserMode so the address that they return to me is not always valid.

That is probably mean as a userspace API.

Calling REQUEST_MMAP and RELEASE_MMAP in the worker does seem to work. I'll have to test the performance of this but at the moment it doesn't seem to be noticeable.

Teaser trailer :)

It's 2:50 in the morning here, but I wanted to share it with you guys. It's working!! :)
It's not ready to be released yet, I need to clanup the code a lot, tune a few things and test it more but if it's not tomorrow then Monday.
So far I listened to half an hour of music at 192kHz, 32bit, 8 channels (downmixed to 2 channels by pulseaudio) and I couldn't hear a single crack in the sound, so it's promising, but the system was not doing much in the background, I'll need to stress test it.

Congrats, and happy Easter :)

Why use IVSHMEM for this? It's a huge hack. QEMU is open source, why not write a virtual PCI device to Qemu specifically for this purpose? As in a VirtIO sound card.

One option would be to specify the IVSHMEM device used by the PCI location triple (bus, address, function). These values are configureable with libvirt and can be read via the Windows API. The Windows application could have default values to look for (bus, address) and the option to configure these values as the user (e.g. commandline parameters)

@gnif You are right, this is a hack, I'm well aware of that. Why do this instead of a fully fledged VirtIO sound card?
A few simple reasons: lack of experience and lack of time on my side, I've been clear on this from the start.
I evaluated both solutions and this seemed to be simpler and faster for me to implement. It's an approach that better match my background and that allowed me to deliver something rapidly.
Usually I have a daily job that's very time consuming, I don't have much free time, but given that we are around Easter and that in Italy we have a lot of holidays between April the 25th and May the 1st I decided to take a route that will allow me to deliver a functional product in this timeframe, before my job will take over again.
In the future I will love to remake this into an independent VirtIO sound card, no doubt about that, but at the moment I simply don't have the knowledge to do so. But I'm studying, and I learn fast.
Two weeks ago when all of this started I had zero experience with windows driver development, no experience with IVSHMEM or VirtIO and no experience with Scream or anything related to this project. But I worked hard night and day to learn what I had to learn and to deliver multichannel support first and then this.
This allowed me to wet my feet in this world, to start working with windows driver, to gain a bit of experience with WinDBG and other useful tools, to see if the ideas I have in my mind are viable or are a waste of time. So far I'm happy with the results and the knowledge I acquired doing this, but I know I can do much much better.
I will learn everything I'll have to learn about VirtIO, Qemu, and whatever, but in the meantime this is what I can do with the resources at my disposal. I hope you'll understand.
The road to a VirtIO driver is long, I'll have to have my code upstreamed by Qemu, and my windows driver signed and distributed. Will Qemu accept my code? Will RedHat / Fedora distribute the driver? MS-PL license allows me to do so? I don't know.
Anyway, my reasons are not important.
Quoting @paul70078 from (here)[https://github.com/gnif/LookingGlass/issues/154#issuecomment-485264895] the thing is allowing more than one IVSHMEM device to be in place without conflict.
What I did here is to loop through the IVSHMEM devices and use the 1st device that have a size defined by the user in a registry key.
If I'll make a patch for looking glass host program to accept an optional command line parameter where the user can specify the size of the IVSHMEM device to use, will you be willing to accept the pull request?

If I'll make a patch for looking glass host program to accept an optional command line parameter where the user can specify the size of the IVSHMEM device to use, will you be willing to accept the pull request?

I have already started on this and @tadly opened at [issue](https://github.com/gnif/LookingGlass/issues/154#issuecomment-485265560 over at Looking Glass).

@paul70078 yes, I quoted you from that issue in my reply. It didn't occur to me that you already started working on a patch. I'll leave it to you then, thank you :)

I think it's time for an Easter present to all of you guys. Scream-IVSHMEM-RC1
See the README.md inside the zip for instructions.

Wow, what can I say - the driver works as expected :+1:
As soon as I remembered that you need Windows test mode for unsigned drivers, I got it installed.
Just run BCDEDIT –Set TESTSIGNING ON in a privileged console, reboot.
The host application updates its configuration for pulseaudio immediately while playing around in the sound driver settings. As promised, CPU usage is near 0%, with no noticeable latency (at least for me). Great job! I will continue testing tomorrow, at different workloads and such.
Happy Easter :)

Thank you @alegru , it's good to know that it's working well on other systems :)
You are right about the test mode for unsigned drivers, I forgot to mention that in the readme.

Hi, I'm testing this new RC. TL;DR lol, it works perfectly. As alegru said that the CPU usage is near a 0%, if you are using 48000Hz 16Bits 6C. Congratulations @martinellimarco and the other people who made this possible.

Thank you @1985a  :) Off topic: I was able to see your blog using a proxy (it doesn't work directly), nice job!

Hi there,  @martinellimarco off topic, Thanks you. Now that you told me you were able to see through a proxy, I realized that a long time ago, when I configure the server, I put in some countries in the ban list. Italy is one of them.
This was made by using ipset through iptables rules.
https://i.imgur.com/n31puxg.png

For those of you that are interested I've just released the ivshmem-alsa receiver. You can get the source code here

"Scream-IVSHMEM-RC1" working great here (Arch, QEMU 4.0, Win10 Guest), 1 MiB seems to be enough even for the highest selectable quality (using 2 channels). No noticeable CPU usage at all :+1: (0% idle, 1% playing, but that's just VLC) Thank you very much!

@PiMaker Thank you for your feedback! :)
Out of curiosity, have you tested the pulseaudio receiver or the alsa one?

@martinellimarco I tested the pulseaudio one. Didn't test the alsa one, and I since had to revert back to the network-based driver since yours is unsigned (which interfers with anti-cheat software, sadly).

Hi everyone, I've merged the code in the master branch.
Expect a new signed release soon, in the meantime I'm closing this issue.
Thanks to all the testers for making this possible.


Renaming .getMetdata() to .metadata()

As discussed in https://github.com/firebase/firebase-ios-sdk/pull/1055 to match the .downloadUrl semantics.

As discussed offline, we think both getMetadata() and metadata() might not be the proper name and would like to give this more thought before making this change.


Enforce HTTPS

HTTPS is needed to use WebUSB, and webcfw.sdsetup.com does not use it. If you try to launch a payload right now it doesn't even let you choose the device.

well spotted

github is in the process of assigning an https certificate, so you'll need to wait for that to happen.

cert is done, https is live. http should redirect to https soon.


app.grammarly.com

@adguard-bot commented on Fri Dec 25 2020
Issue URL (Anti Adblock Script)
https://app.grammarly.com/diagnostic-test?origin=editor
Comment
Username: @sandervdbroekisd

Whenever I use grammarly using safari with adguard enabled it has issues with connectivity. It starts working again when I disable adguard (system level, using the browser plugin does not seem to affect it). Grammarly does work with Chrome
The information below is automatically generated. Do not delete it.
Referrer:
https://app.grammarly.com/ddocs/807877972
Custom userscripts:
https://userscripts.adtidy.org/release/adguard-extra/1.0/adguard-extra.meta.js

Screenshots

Screenshot 1

System configuration

Information
value

AdGuard product:
AdGuard for Mac v<IP_ADDRESS>8 release

Browser:
Safari

Filters:
Ad Blocking: AdGuard BasePrivacy: AdGuard Tracking ProtectionSocial Widgets: AdGuard Social MediaLanguage-specific: AdGuard German, AdGuard Japanese, AdGuard Dutch, AdGuard Spanish/Portuguese, AdGuard French, AdGuard Chinese, EasyList Italy, Liste AR, Icelandic ABP List, AdBlockID

Custom filter subscriptions:

@stale[bot] commented on Sun Jan 10 2021
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@sandervdbroekisd just in case, do you use MacOS Big Sure or an older version?

@sandervdbroekisd just in case, do you use MacOS Big Sure or an older version?

Also, do you use Little Snitch or anything like that?
There's a known bug in the current Big Sur version due to which AdGuard and Little Snitch have compatibility issues. The bug causes exactly this kind of connectivity problems when there are two network extensions registered in the system. Note, that the extensions don't even have to do anything with the connections.
Apple is aware of the issue. We reported it when Big Sur was first released, and I suppose Little Snitch reported it too so I hope we can expect a fix in one of the upcoming Big Sur patches.

Also, do you use Little Snitch or anything like that?
There's a known bug in the current Big Sur version due to which AdGuard and Little Snitch have compatibility issues. The bug causes exactly this kind of connectivity problems when there are two network extensions registered in the system. Note, that the extensions don't even have to do anything with the connections.
Apple is aware of the issue. We reported it when Big Sur was first released, and I suppose Little Snitch reported it too so I hope we can expect a fix in one of the upcoming Big Sur patches.

Hi @ameshkov I do use Big Sur, I don't use Little Snitch or any other third-party firewall. I do use openVPN and Cisco connect VPN clients, I don't know if they could cause similar issues?
I managed to get Grammarly working by switching to "automatic proxy" mode, though this is obviously not an ideal solution.

Hi @ameshkov I do use Big Sur, I don't use Little Snitch or any other third-party firewall. I do use openVPN and Cisco connect VPN clients, I don't know if they could cause similar issues?
I managed to get Grammarly working by switching to "automatic proxy" mode, though this is obviously not an ideal solution.

Cisco connect can indeed cause this, but unlike the LS issue we haven't yet reported that to Apple.

Cisco connect can indeed cause this, but unlike the LS issue we haven't yet reported that to Apple.

https://www.cisco.com/c/en/us/td/docs/security/vpn_client/anyconnect/anyconnect49/upgrade/AnyConnect_macOS_BigSur_Advisory.html
According to this, they setup content filter too. They are also set up app/transparent proxy, so it may interfere with AdGuard even when bug will be fixed.

https://www.cisco.com/c/en/us/td/docs/security/vpn_client/anyconnect/anyconnect49/upgrade/AnyConnect_macOS_BigSur_Advisory.html
According to this, they setup content filter too. They are also set up app/transparent proxy, so it may interfere with AdGuard even when bug will be fixed.

Unfortunately, Cisco Connect is not compatible with the AdGuard app.
It uses the same socket filtering method as the AdGuard Network extension, so they can't work simultaneously


Fixes: Field  overridden by a non-annotated attribute.

I have this error:
: Field 'message_type' defined on a base class was overridden by a non-annotated attribute. All field definitions, including overrides, require a type annotation
I think that this patch corrects it.

Welcome to ReVa! The Reverse Engineering Assistant
Logging to /tmp/ReVa-20240202-190943.reva.log
Traceback (most recent call last):
File "/home/velteyn/venv/bin/revassistant", line 8, in
sys.exit(main())
^^^^^^
File "/home/velteyn/venv/lib/python3.11/site-packages/reverse_engineering_assistant/assistant.py", line 378, in main
from .assistant_api_server import run_server
File "/home/velteyn/venv/lib/python3.11/site-packages/reverse_engineering_assistant/assistant_api_server.py", line 23, in
from .tool_protocol import RevaGetDataAtAddress, RevaGetDataAtAddressResponse, RevaHeartbeat, RevaHeartbeatResponse, RevaMessageResponse, RevaMessageToReva, RevaMessageToTool, RevaMessage
File "/home/velteyn/venv/lib/python3.11/site-packages/reverse_engineering_assistant/tool_protocol.py", line 252, in
class RevaGetDecompilationResponse(RevaMessageToReva, RevaMessageResponse):
File "/home/velteyn/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 92, in new
private_attributes = inspect_namespace(
^^^^^^^^^^^^^^^^^^
File "/home/velteyn/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 362, in inspect_namespace
raise PydanticUserError(
pydantic.errors.PydanticUserError: Field 'message_type' defined on a base class was overridden by a non-annotated attribute. All field definitions, including overrides, require a type annotation.
For further information visit https://errors.pydantic.dev/2.5/u/model-field-overridden


[PR][CLOSED] Added comment for plugins audit configuration

 Issue by geunsik-lim
Tuesday Jul 03, 2018 at 00:15 GMT
Originally opened as https://github.sec.samsung.net/STAR/TAOS-CI/pull/78

This PR is to append a comment for plugins audit configuration.
Signed-off-by: Geunsik Lim<EMAIL_ADDRESS>
geunsik-lim included the following code: https://github.sec.samsung.net/STAR/TAOS-CI/pull/78/commits

:octocat: cibot: Thank you for posting issue #102. The person in charge will reply soon.

 Comment by git-bot-sec
Tuesday Jul 03, 2018 at 00:15 GMT

Thank you for submitting PR #78. Your PR has to pass all verificiation processes of cibot before getting a review process from reviewers. We recommend that you read instruction manual in . If you want to monitor the progress status of your PR in detail, visit http://aaci.mooo.com/.

 Comment by git-bot-sec
Tuesday Jul 03, 2018 at 00:15 GMT

:octocat: cibot: @geunsik-lim, One of the format checkers is failed. If you want to get a hint to fix this issue, please go to https://github.sec.samsung.net/STAR/TAOS-CI/wiki/.


[Bug 1987271] Requeuing ManagedOCS CR upon conflict while uninstall

While updating the status for the ManagedOCS cr the request is stale and it was causing a panic. This PR attempts to solve the issue by requeuing the request after 10 sec.

@ezio-auditore I don't get the fix. Returning an error will ensure a queue (and a log entry) so why do you propose to change that and how does it solve the panic?


v8 5.1.281.47 issue with Nan::Null()

Running electron 1.2.0, which bundles v8 5.1.281.47 (node 6.2.0 bundles v8 <IP_ADDRESS>), i hit an issue where this code from leveldown (https://github.com/Level/leveldown/blob/master/src/database_async.cc#L141):
  v8::Local<v8::Value> argv[] = {
      Nan::Null()
    , returnValue
  };
  callback->Call(2, argv);

calls the js callback with a boolean true as the first parameter.
Steps to reproduce:
create app.js with:
var db = require('level')('/tmp/db');
db.db.put('foo', 'bar', console.log.bind(console));
db.db.get('foo', err => console.log('err', err));

then
$ nvm use 6
$ npm install<EMAIL_ADDRESS>level
$ node_modules/.bin/electron app.js

This error does not happen with electron 1.1.3, which is bundling v8 <IP_ADDRESS>.

Can't reproduce on windows. Did you run electron-rebuild?

@mathiask88 sorry, my reproduction steps were missing a few details. can you try again?

@juliangruber I meant that I could reproduce your code but not the error :) I ran the electron-starter repository with electron-prebuild 1.2.0 and level 1.4.0 and I get null. bar

@mathiask88 hm, then it looks like this issue doesn't on windows I'd say

@juliangruber Started my Ubuntu VM and it seems that electron-rebuild does not rebuild leveldown even if I use the --force argument. electron-rebuild does not have a verbose option. No NAN issue in my opinion. Maybe an error with prebuild? Because for windows there are no prebuilds available and it is necessary to compile.

@mathiask88 the problem persists when i manually rebuild leveldown. you shouldn't need to rebuild leveldown though if you use node 6 when installing, as both node 6 and electron 1.1.3/1.2.0 have abi 48.
my problem literally disappears by issuing npm install<EMAIL_ADDRESS>and rerunning app.js.

V8 5.1 is not ABI compatible with V8 5.0. Electron should not be shipping
this with ABI 48.
On May 31, 2016 6:35 AM, "Julian Gruber"<EMAIL_ADDRESS>wrote:

@mathiask88 https://github.com/mathiask88 the problem persists when i
manually rebuild leveldown. you shouldn't need to rebuild leveldown though
if you use node 6 when installing, as both node 6 and electron 1.1.3/1.2.0
have abi 48.
my problem literally disappears by issuing npm install
<EMAIL_ADDRESS>and rerunning app.js.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/nan/issues/578#issuecomment-222689741, or mute
the thread
https://github.com/notifications/unsubscribe/AAE0qelTMBzhxn1t6anBjKnbKNdP8YsUks5qHDkEgaJpZM4IqeQu
.

@juliangruber Can you try this patch on top of the version of V8 5.1 included by electron: https://github.com/ofrobots/node/pull/23 and see if that solves the problem?

Seems like this issue breaks most of the level projects. /cc @rvagg

as both node 6 and electron 1.1.3/1.2.0 have abi 48

Wait, what?? they are shipping 5.1 and haven't changed ABI, I thought they went with a >100 number cause they were doing things differently.

as both node 6 and electron 1.1.3/1.2.0 have abi 48

This is the V8 that's linked to the 1.2.0 Electron tag (via node) https://github.com/electron/node/blob/d4528c219df8f442d769bae054883e1af79f105e/deps/v8/include/v8-version.h and it says 5.0. How are you determining that you have 5.1 @juliangruber?

Because it does not build Node against the bundled V8, but downloads whatever Chromium was built with and links against that. All of this is rather messed up, to say the least, but none of it pertains to NAN, so I am closing this as invalid.

thanks @kkoopa, that def. makes sense. I opened an issue on electron here: https://github.com/electron/electron/issues/5851


Provide an API to use Filetype icons

Currently you can use the product icons as codicons, but there seems to be no way to easily use the filesystem icons. Provide an API to get filetype icons in treeviews and webviews.

I tree views you can get file type icons by adding a resourceUri to your tree items with the file extension of the file type icon you want.

The described behavior is how it is expected to work. If you disagree, please explain what is expected and what is not in more detail. See also our issue reporting guidelines.
Happy Coding!

And what about webviews?


fix: Rename  to 'com/example' to 'com/scate'.

Closes #1

it's now fixed and 0.2.1 is released, thanks for the effort


Unit test for the single speed/stage IEER calculation

new json file and new unit tests for DX.

@wanhanlong1130 - Could you please do a reverse merge of the develop branch into your feature branch? It looks like there are a few conflicts.

I've linked issues that this PR should address. Let's make sure that they're addressed before merging it!

@aowabinr can you please review the library file brought over from #95? Thanks!

@wanhanlong1130 - Could you please do a reverse merge of the develop branch into your feature branch? It looks like there are a few conflicts.

Already done!


V0.0 ChuaPingChan

Coverage remained the same at 89.093% when pulling 42f2cbc637f93fa41b187066a4e2928fddf68185 on V0.0_pingchan into ffff338cb58fdc6e92482ee3bd379155c76295ed on V0.0.


VP9 Videos crash VideoPlayer on Windows

Godot version:
Tested on 3.2.1 and 3.2.2 rc1

OS/device including version:
Windows 10
Issue description:
When a VP9 webm video is played the program crashes after about 0.1-0.2 seconds without showing any video.
A VP8 encoded file works and the VP9 encoded files works on the Linux and Android version without crashing
Steps to reproduce:
Scene Tree
Root
  Viewport
    Control
      Videoplayer
  MeshInstance (with Viewport Texture)

Code
var stream = VideoStreamWebm.new()
stream.set_file("res://assets/crunch_sample.webm")
get_node("Viewport/Control/Videoplayer").stream = stream
get_node("Viewport/Control/Videoplayer").play()

Minimal reproduction project:
videotest.zip

Duplicate of #37648.


Format curated collection of existing training notebooks

https://github.com/brightway-lca/brightway-training/commit/34d2de34f77feecb542f7b8da1440a2fc298f043 added draft of curated list of Brightway training notebooks originally created by @KarinTreyer et al. during the 2022 Brightcon Hackathon on Documentation.

...compare: https://github.com/brightway-lca/notebook-collection


Cant setup Dependency Proxy from gitlab

I can't setup k3s to download container via gitlab dependency proxy.
https://docs.gitlab.com/ee/user/packages/dependency_proxy/
    configs:
      registry.example.com:
        auth:
          username: token
          password: greg34g34g545g54
    mirrors:
      docker.io:
        endpoint:
          - "https://registry.example.com"
        rewrite:
          "^(.*)": "company/dependency_proxy/containers/$1"

I received logs:
E0724 01:19:30.513755      60 kuberuntime_manager.go:815] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = fail │
│ ed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack │
│  image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": pull access den │
│ ied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" pod="kube-syste │
│ m/local-path-provisioner-7b7dc8d6f5-5g27d"                                                                                │
│ E0724 01:19:30.513937      60 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"local-path-p │
│ rovisioner-7b7dc8d6f5-5g27d_kube-system(b5421ea0-48a1-4906-907b-83a847f282b1)\" with CreatePodSandboxError: \"Failed to create sandbox │
│  for pod \\\"local-path-provisioner-7b7dc8d6f5-5g27d_kube-system(b5421ea0-48a1-4906-907b-83a847f282b1)\\\": rpc error: code = Unknown  │
│ desc = failed to get sandbox image \\\"rancher/mirrored-pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed │
│  to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pau │
│ se:3.6\\\": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorizat │
│ ion failed\"" pod="kube-system/local-path-provisioner-7b7dc8d6f5-5g27d" podUID=b5421ea0-48a1-4906-907b-83a847f282b1

Check the containerd log file; it frequently contains more detailed information on pull failures. I suspect that there is some problem with the pull-through cache and it is falling back to docker hub.


How to catch caret-position-changed event?

hi, dear jacobslusser!
I wanna do something when user move the caret in scintilla. How can I catch the event?
Look forward to your guidance. Thank you!

The UpdateUI even is what you want.

Thanks for your help. I am designing a text editor based on ScintillaNet. The  editor is very like editor in Visual Studio. I'd like to share my code when finnishing.

UPDATEUI handlers are called after the document is painted, so the style changing methods called inside the handler will get effect in the next paint phase.


Add wild fire setting switches

This pull request adds several switches to the config file for granular control of wildfire behavior.  The PR is being filed as a draft because changes were made on master before I finished work that overlap some functionality (see XX_dsb notes below).  I would like some feedback on how to revise these code changes to fit with XX_dsb additions.
There are some comments and other know issues in the changes that will be cleaned up prior to the final pull request.
Purpose:
These changes have four main purposes.

Make it possible to turn fire on an off with settings rather than changing variable values in the code.
Make it possible to turn fire on and off for different run stages.
Make it possible to use Fire Return Interval (FRI) driven fire for transient and scenario stages.
Do some preparation for upcoming extensions to the wildfire implementation.

Details:
Fire Switches for Run Stages:
The config file settings fire_on_PR, fire_on_EQ, fire_on_SP, fire_on_TR, and  fire_on_SC have been added.  Each allows fire to be turned on for each of the corresponding run stages.  Together these achieve purposes 1 and 2 above.
I have used values 0 for 'off' and 1 for 'on'.  Currently there is no reason they could not be true and false, but I was leaving room for expansibility, as further mode information could be provided in the future.  It is important to note that 'on' only means wildfire is possible, not that it will occur.  Wildfire will occur only when the fire is specified to occur.  The input files that govern fire have not been modified.
I have opted not to include a global fire switch that would override the above switches.  Fire is turned off when all fire_on_XX switches are set to 0.  A master switch could be convenient but I decided it would potentially lead to confusion.  It could be added.
Outstanding Issues:
@hgenet says that fire does not occur in the pre-run stage.  However, the code suggests otherwise so I'm not sure.  If fire should not occur in the pre-run then fire_on_PR will be removed.  As is there is no way to test if fire is running in the pre-run stage anyway.  Feedback would be appreciated on this issue.
The XX_dsb settings that have been recently added partially duplicate the functionality of the fire_on_XX switches.   Or at least they did.  When I merged some changes a while back I had a conflict because these settings were being used in the fire logic.  Now it seems those issues have gone away (actually I think the issue was in TEM.cpp, see Additional Technical Details)?  If anyone knows the history here please weigh in.  Fire is currently the only disturbance (dsb) but that may not always be the case.
Fire Mode Switch:
The config file setting fire_ignition_mode has been added.  The default value of 0 makes the model run in the traditional way.  FRI based fire is used for the equilibrium and spin-up stages and explicit fire for the transient and scenario stages.
Setting fire_ignition_mode = 1 causes FRI based fire to be used for the transient and scenario stages as well (purpose 3 above).  The fri-fire.nc input file is used in the same manner as in the other run stages to control fire occurrence.
Work is ongoing on a revised wildfire implementation.  I want to maintain the current/original/traditional wildfire behavior for backward compatibly and comparison purposes.  New values for fire_ignition_mode will be added in the future to allow new functionality while preserving the existing.
The name fire_ignition_mode, while currently reasonably descriptive, might need to be renamed the governed fire behavior may extend beyond ignitions.
Namespacing:
The 'fire_' prefix has been used for all switches.  It is likely that several additional wildfire related settings will be added in the future and I wanted to create a clear namespace for them.  I placed the switches in the model_settings section of the config file because I thought that is where fit.  This differs from the approach of the 'XX_eq' settings.  Perhaps there could be a fire section?  Any thoughts?
Model Behavior Changes:
In addition to the changes listed above there is one small change to fire behavior.  Previously if a FRI = 0 value was specified for a grid cell that was simulated a divide by zero exception would occur and cause a crash.  I have modified the logic so that FRI = 0 means no fire will occur for that grid cell.  While the divide by zero is clearly a bug I suspect that is exists because FRI based fire has historically been been specified as non-zero for most simulations.  It makes sense to me that 0 represents no fire return interval but others might disagree.  This code change could be considered armoring of the code of a feature addition depending on your perspective.  FRI = 0 can cause another problem in the equilibrium phase.  See below.
Testing:
This code has been debugged and tested at 3057e2b0.  Simulations were performed with may different combinations of fire_on_XX and fire_ignition_mode settings.  The switches were tested for the equilibrium, spin-up, and transient modes.  The pre-run was not tested for the reasons discussed above.  The scenario stage was also put off because the behavior is identical to the transient stage and because a new round of testing will be required after the issues discussed here are resolved.  Fire was successfully turned on and off by stage for runs using identical fire input files.  FRI based fire was confirmed for the transient stage.
An issue came up in testing with variable FRI values.  For spin-up and transient stages having different FRI lengths works fine.  I encountered problems in the equilibrium stage however.  As is documented in the code the spin-up needs to start before the end of a fire disturbance cycle.  When the FRI lengths vary, and included a grid cell with FRI = 0, this caused a crash trying to recalculate the equilibrium run length.  I have not investigated whether this is just because of the zero run length or the because of the variability.  I'm not sure why this requirement is there so I didn't want to spend much time on it.  This is not a issue created by my changes per se, but it makes a concise set of test simulations difficult.
Documentation:
Documentation is only inline currently.  Documentation will be added to the documents when the issues discussed above are resolved.

Test Simulations:
Here are a selection of test simulations.  I can post more if requested.  All simulations were run for 4 grid cells from PR to TR stages.
Test Set 1:
In this set of simulations equilibrium, spin-up, and transient stages were run for 110 years.  Fire was turned on for the spin-up and transient stages.  FRI lengths of 0, 30, 40, and 50 years were used for the four grid cells.  For explicit fire grid cells had 0, 1, 3, or 5 fires (on every 4th prime year). Severity varies with grid cell, but this is not really relevant.  They plots show the effect of changing fire_ignition_mode from 0 to 1.

Test Set 2:
In this set of simulations equilibrium was run for the default 1000 years.  The spin-up and transient stages were run for 210 years.  FRI was set to 100 years for all grid cells.  There is no evidence of fire in cell 1,1 because the severity = 0.  Explicit fire setting were the same as above.  Three plot showing the effect of fire_ignition_mode and the ability to control fire across run stages with the fire_on_XX switches are shown.

Thanks Joshua for the very well documented request. I'll try to read it before the end of the week.

re: the history of the conflict between XX_dsb and the fire_on_EQ flags is that your changes started before we merged PR #628, so they overlap.
re: turning fire on in pre-run, there is no use case for this but it would technically be possible to turn the flag on. The ecosystem would probably crash quickly. There is really no need to have the setting exposed for the pre-run stage, but it makes the files more consistent with everything else :shrug:
re: FRI=0 being no fire, that makese sense. Probably that should get documented in a section describing the input files (maybe "Model Overview" > Inputs/Outputs" > "Inputs"?) https://uaf-arctic-eco-modeling.github.io/dvm-dos-tem/model_overview.html#inputs
re: model behavior changes (FRI = 0 in equlibrium phase), it could be a divide by zero problem or could be the problem described in issue #613.
re: as for what to do about merging this PR, got to talk with @hgenet and @rarutter about the overall design and are thinking that a config format to aim for would look like this:
"stage_settings": {
  "pre-run": {
    "dsl": false,
    "fire": false,
  },
  "eq": {
    "dsl": true,
    "fire": true,
  },
  // ... etc
},
"module_settings" {
  "dsb": {
    "fire": {
      // 0: default/old, 1: use fri for this stage, 2: future
      "ignition_pr":
      "ignition_eq":
      "ignition_tr":
      "ignition_sp":
      "ignition_sc":

      // more future controls per staage can go here...
      "severity_pr":
      "severity_eq":
      // ...
    },
    "insect": { ... },
    "thermokarst":{ ... },
  }
},
"model_settings" : { }

Would this accomodate your anticipated needs?
We are not sure how many hoops to jump thru for merging this current PR, but we'd like to aim towards the design laid out above. One way to move forward would be to effect the entire refactor right now (more commits to this PR). Another would be to implement only parts (i.e. renaming the config stage_settings dsb --> fire and the ModelData.dsbmodule member), but not all the functions related to dsb at this time. This would leave the code in a slightly more confusing state (confusing names) but would be much less work upfront. If we go this route we would want to leave comments in the code explaing why the name dsb is used in some places and fire in others so that our future selves don't go crazy.
If time permits it might be better to fully flesh out this design and undertake the whole refactor now. If merging the PR sooner would greatly facilitate your work we could defer some of the refactoring and just button up a few small details.

Sorry for the long silence on this PR.  I'm planning on putting some time into this in order to resolve the remaining issues over the next couple of weeks.  I'm providing an item by item responses to @tobeycarman's comments below but in short I think the proposed config file compromise will work for me and I can work on implementing the revisions as part of this PR (with one question below).  A Zoom discussion would probably be good to make sure I'm following everything said above.  I've started pulling updates from master to my local repo and at some point those will be pushed back here with further edits.  It might be a few commits before any further attention is needed.  I yell when I get to another stable version.
Specific responses:

re: turning fire on in pre-run, there is no use case for this but it would technically be possible to turn the flag on...

I don't want to change behavior in the pre-run so I am removing the switch in the config file.

re: FRI=0 being no fire, that makese sense. Probably that should get documented in a section describing the input files...

I will work on updates to the documentation to cover this and tbe other changes (once finalized).  Is is preferred to documentation changes as port of a code PR or as a separate one?

re: model behavior changes (FRI = 0 in equlibrium phase), it could be a divide by zero problem or could be the problem described in issue #613.

I'm familiar with issue #613 and this should be separate.  My code fix seems to be working but we can reassess if any issues come up in subsequent testing.

re: as for what to do about merging this PR, got to talk with @hgenet and @rarutter about the overall design and are thinking that a config format to aim for would look like this:

I'm am fine with the solution proposed with separate switches to turn fire on and off for each run stage and dsb:fire switches to control fire run mode.  ignition_XX is probably a fine terminology for the these switches.  I anticipate additional fire mode switches may be needed fairly soon so this namespace will work nicely.

We are not sure how many hoops to jump thru...

I think I follow this but a Zoom discussion (per above) might help.  I'm glad to help with the code changes if I'm sure I know what you want.


FIX #4419 : added margin support in text widget

Description

Added margin support in text widget.

Fixes #4419
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Tested Locally.

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

Test coverage results :test_tube:
:red_circle: Total coverage has decreased
// Code coverage diff between base branch:release and head branch: FEATURE/4419-text-widget-padding-controls

Status
File
% Stmts
% Branch
% Funcs
% Lines

:red_circle:
total
53.84 (0)
35.81 (0)
32.43 (-0.01)
54.42 (0)

:red_circle:
app/client/src/components/designSystems/blueprint/TextComponent.tsx
41.18 (-2.57)
10 (0)
8.33 (-0.76)
39.39 (-2.55)

/ok-to-test sha=2545bc2

@vibhandikyash If I use bigger padding then the text is displayed out of the widget due to this it overlaps the below widget.

@marks0351 any solutions around this?

@marks0351 any solutions around this?

we should be decreasing the height of widget based on the padding added. widget content should not be overflowing its borders it can overflow within it selves like we do with containers.

@somangshu should we temporarily close this PR?

@vibhandikyash please have a discussion with @marks0351 and take this forward.

@marks0351 As we discussed. This is dependent on #2587. Once that is merged. We can take this further.
cc: @somangshu

This will be picked up after the container widget branch is merged, Closing this for now


US 02.03.01

As an attendee, I want to receive push notifications with important updates from the event organizers.
Acceptance Criteria

[x] Attendees can easily opt-in to receive push notifications for events they are interested in or registered for.

[x] Attendees have control over the types of notifications they receive, allowing customization based on their preferences.

[x] Attendees receive notifications in real-time, ensuring they are informed of important updates, changes, or announcements related to the events.

[x] Notifications are clear, concise, and provide essential information, minimizing confusion and information overload.

[x] Where applicable, notifications include direct links to where attendees can find more detailed information within the app.

Completed


Discontinue using RWX PVs.

Many users clusters (especially in cloud) don't provide storage classes that support RWX.
Solution: The addons will download/upload files (trees) instead.  The maven optimization provided by sharing the m2 cache using RWX can likely be accomplished by using a caching proxy (like squid).
To support this, the bucket API will need to be augmented to support GET/POST/PUT of file trees. Each bucket owner endpoint will handle directories by tar.gz files.  The combination of detecting the URL path points to a directory and the use of Content-Type and Accept: headers will guide the API.
Buckets
GET /tasks/1/bucket/rules (rules is a directory of rule files)

When: Accept: will stream back index.html.
When:  Accept:application/x-gtar will build a tar.gz of the directory and stream it back with Content-Type:application/x-gtar

PUT /applications/1/bucket/windup/report (report is a director of: windup html files)

When: Content-Type: Store the .tar.gz (body) as a file at the specified path.
When: Content-Type:application/x-gtar Replace the file/directory at the specified path with the content of the tar.gz (body).

Maven Cache
Perhaps we could deploy a squid cache in a pod and somehow inject its URL into the settings.xml?

Confirm I'm working on the tar.gz GET&PUT actions that could be reused for this issue. I'd have few questions just to be sure.
Using a "sub-path" within bucket owner endpoint - agreed :+1:  - this assumes that the owner has just one bucket - is this going to be right also in close future?
The approach using Accept/Content-Type headers looks reasonable to me :+1:
Not critical, but I was looking on mimetypes used for downloading tar.gz files, it looks the typically used mimetype is application/x-gzip (gin gonic sets it when sending a tar.gz file, kernel.org sends it for compressed tar with Linux kernel sources). Could the mimetype be  application/x-gzip at least for a beginning or open to be discussed? (application/x-gtar looks to me to represent a tar somehow compressed, not necessary using gzip)
If we used a caching proxy, we might need implement HEAD or similar method to provide headers saying if given archive is up-to-date (like Last-Modified based on modification time of files in the bucker directory).

Confirm I'm working on the tar.gz GET&PUT actions that could be reused for this issue. I'd have few questions just to be sure.

Yes.

Using a "sub-path" within bucket owner endpoint - agreed +1 - this assumes that the owner has just one bucket - is this going to be right also in close future?

Yes.

Not critical, but I was looking on mimetypes used for downloading tar.gz files, it looks the typically used mimetype is application/x-gzip (gin gonic sets it when sending a tar.gz file, kernel.org sends it for compressed tar with Linux kernel sources). Could the mimetype be application/x-gzip at least for a beginning or open to be discussed? (application/x-gtar looks to me to represent a tar somehow compressed, not necessary using gzip)

Fine with whatever is appropriate.

If we used a caching proxy, we might need implement HEAD or similar method to provide headers saying if given archive is up-to-date (like Last-Modified based on modification time of files in the bucker directory).


Feature request: GaLore optimizer

Feature request for the Gradient Low-Rank Projection (GaLore) optimizer.
The GaLore optimizer computes low-rank gradients way to dramatically reduce memory. The ArXiv paper is here and the original Github implementation is here.
GaLore is quite popular: HuggingFace has implemented it here, PyTorch Lightning is trying to implement it here. The original implementation has 1K+ Github stars.
A good starting point may be this clean PyTorch implementation here in the PyTorch Optimizers library.
Lastly, readers should also be aware of the improved Q-GaLore paper and repository here and here respectively.

Hello @gil2rok,
Thanks for pointing this out! Would you be willing to contribute with an implementation?
If not, we'll add it to our todo list but any contributions are super welcome! :)

At the moment, I do not have the bandwidth to add this optimizer. If I have some unexpected time, will try to come back to this cause I think it'd be super cool to implement.
Thanks for all you guys do for this library!

Hi All,
I am working with my colleague and we have a working prototype of Galore implementation. We are doing some final tests. Can you suggest if we should follow PR diff. Please point us to any clear guidelines. super appreciate the help.

That's great news!
Yes, the workflow currently would be to

add this optimizer in the contrib folder, note that the PR you pointed out introduced some errors (like in the dtypes at initializations), so take a look at the latest version of e.g. prodigy or schedule_free.
add this optimizer to the common_test of this folder, in particular making sure it can support some of optax functionalities like MultiSteps.
if some specific functionalities of this optimizer requires some tests, feel free to add a file that make these tests (schedule_free for example has its own specific tests)

Feel free to start a PR and send it to us so that we can help reviewing it and integrating it.
Thank you for this!


[flutter_tool] Use a timeout for xcode showBuildSettings

Description
Subcommands called by the tool may hang. Subsequent invocations may then succeed. This PR adds optional named parameters to runAsync() and runCheckedAsync() to provide for the need to retry subcommands that time-out.
In particular, when xcode runs with the -showBuildSettings flag it may timeout and then succeed on retry. This PR updates the call site from the issue reported below to time out after one minute, and retry one time. Additionally, this PR caches the result of the -showBuildSettings subcommand that could not be made asynchronous since it is called from a constructor.
Related Issues
https://github.com/flutter/flutter/issues/35988
https://github.com/flutter/flutter/issues/39141
Tests
I added the following tests:
Added tests of timeout with retry to process_test.dart. Updated other affected tests.
Breaking Change
Does your PR require Flutter developers to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (Please read [Handling breaking changes]). Replace this with a link to the e-mail where you asked for input on this proposed change.
[x] No, this is not a breaking change.

This violates our style guide which says to never have timeouts because they are a source of flakes.
Are these processes destructive or idempotent? If they're idempotent, then I suggest that we change this to not kill the first process, but to show a message after a few seconds saying "This seems to be taking longer than usual" and then start a second process in parallel, to see if it returns faster, and if it does, kill the first one. If the second takes more than a minute, kill it then try a third one, etc, leaving still the first one running, in case it's just being super slow.

Filed as https://github.com/flutter/flutter/issues/42265