Image orientation wrong

Hi, thanks for the great lib - it helps me so much.
I have a problem with some images - the are inserted in a wrong orientation, and then the aspect ration is not correct of course. How can I fix it before i insert an image? I use following code:
insertImgFit - function in additionalJsContext:
var data=fs.readFileSync(filename);
var ext = path.extname(filename).toLowerCase();
[some calculation of width and height - no problem here]
return { width: bw, height: bh, data, extension: ext };

The image is inserted with the correct width and height - but 90 degrees rotated. I think I have to check somehow the EXIF Orientation flag an rotate the image - does someone know how this can be done?
I am using the lib in nodejs, on mac and on windows.

I found a solution that I want to share - I used 'jpeg-autorotate':
const autorotate = require('jpeg-autorotate');

insertImgFit: async function(id, w, h) {
   var data2=fs.readFileSync(imgObj.img);
   var {buffer, orientation, dimensions, quality} = await autorotate.rotate(data2, {quality:99});
   var ext = path.extname(imgObj.img).toLowerCase();
   return { width: bw, height: bh, data:buffer, extension: ext };
}

the async/await was the key I was searching for ...

I found a solution that I want to share - I used 'jpeg-autorotate':
const autorotate = require('jpeg-autorotate');

insertImgFit: async function(id, w, h) {
   var data2=fs.readFileSync(imgObj.img);
   var {buffer, orientation, dimensions, quality} = await autorotate.rotate(data2, {quality:99});
   var ext = path.extname(imgObj.img).toLowerCase();
   return { width: bw, height: bh, data:buffer, extension: ext };
}

the async/await was the key I was searching for ...

Thanks for posting your solution!
On Fri, 29 Jan 2021 at 19:18, trx222<EMAIL_ADDRESS>wrote:

Closed #189 https://github.com/guigrpa/docx-templates/issues/189.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/guigrpa/docx-templates/issues/189#event-4267433326,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC37OQHB2PHPJXBCJIMF5J3S4L3YZANCNFSM4WZFZACA
.

Thanks for posting your solution!
On Fri, 29 Jan 2021 at 19:18, trx222<EMAIL_ADDRESS>wrote:

Closed #189 https://github.com/guigrpa/docx-templates/issues/189.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/guigrpa/docx-templates/issues/189#event-4267433326,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC37OQHB2PHPJXBCJIMF5J3S4L3YZANCNFSM4WZFZACA
.


Fix a memory leak in CircularBuffer

Add a destructor to properly clean up the buffer.
new only returns a nullpointer as expected in the surrounding function if std::nothrow is used.

Looks good!


(v2.0.0-beta5) Failed to add swap (swapon failed: Invalid argument)

BurmillaOS Version: (ros os version)
v2.0.0-beta5
Where are you running BurmillaOS? (docker-machine, AWS, GCE, baremetal, etc.)
a cloud service (kvm based hosting)
Which console you use (default, ubuntu, centos, etc..)
default
Please share copy of your cloud-init (remember remove all sensitive data first)
write_files:
- content: |+
    #!/bin/sh
    # add swap (taken from https://web.archive.org/web/20170713205503/https://coreos.com/os/docs/latest/adding-swap.html)
    SWAP_PATH=/var/vm
    SWAP_FILE=swapfile1
    SWAP_SIZE=4g
    SWAPINESS=5
    /bin/rm -rf ${SWAP_PATH}
    /bin/mkdir -p ${SWAP_PATH}
    /bin/touch ${SWAP_PATH}/${SWAP_FILE}
    /bin/bash -c "fallocate -l ${SWAP_SIZE} ${SWAP_PATH}/${SWAP_FILE}"
    /bin/chmod 600 ${SWAP_PATH}/${SWAP_FILE}
    /sbin/mkswap ${SWAP_PATH}/${SWAP_FILE}
    /sbin/sysctl vm.swappiness=${SWAPINESS}
    /sbin/swapon ${SWAP_PATH}/${SWAP_FILE}
  owner: root
  path: /etc/rc.local
  permissions: "0755"

Thank you for developing version 2.
I tried v2.0.0-beta5, and found a problem with adding a swap.
rancher@myhost*:~:-) sudo /etc/rc.local                                                                                (Aug 30) 13:54:26
Setting up swapspace version 1, size = 4 GiB<PHONE_NUMBER> bytes)
no label, UUID=dc934851-c7b2-4b4b-93ba-df4b6cd6e55a
vm.swappiness = 5
swapon: /var/vm/swapfile1: swapon failed: Invalid argument
rancher@myhost*:~:-(

I googled about this problem, and found some sites that say its cause is sparse file handling.

[SOLVED] Swap not working with Kernel 5.7.2 / Kernel & Hardware / Arch Linux Forums
FS#66921 : [linux] "Swapfile has holes" on Linux 5.7

So, I tried making a swap using dd, but no luck.
rancher@myhost*:~:-( sudo rm /var/vm/swapfile1

rancher@myhost*:~:-) sudo dd if=/dev/zero of=/var/vm/swapfile1 bs=4M count=1024 status=progress
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.86539 s, 546 MB/s
1024+0 records in
1024+0 records out

rancher@myhost*:~:-) sudo chmod 600 /var/vm/swapfile1

rancher@myhost*:~:-) sudo mkswap /var/vm/swapfile1
mkswap: /var/vm/swapfile1: warning: wiping old swap signature.
Setting up swapspace version 1, size = 4 GiB<PHONE_NUMBER> bytes)
no label, UUID=93566d66-a07e-4db4-a437-0c4e7785b319

rancher@myhost*:~:-) sudo swapon -v /var/vm/swapfile1
swapon: /var/vm/swapfile1: found signature [pagesize=4096, signature=swap]
swapon: /var/vm/swapfile1: pagesize=4096, swapsize=4294967296, devsize=4294967296
swapon /var/vm/swapfile1
swapon: /var/vm/swapfile1: swapon failed: Invalid argument

Swap is mandatory for me.
Could you confirm this?
p.s.
I tried v2.0.0-beta5 almost the whole day, I don't have any problem other than this issue.

I would try by having separate partition or disk for swap and by following https://burmillaos.org/docs/storage/custom-partition-layout/#burmilla_swap instead of partition file.
Also note that we looks to have search & replace failure on that guide, those partition labels should use prefix RANCHER_ instead of BURMILLA_ for backward compatibility.

Umm, I prefer swapfile.
It's pretty handy because it can be configurable within cloud-init.yml only.
I don't want to change the disk layout on every install.
I've perhaps read v2 kernel shares k3os's kernel code, somewhere.
Do you think this problem exists on k3os, too? (I don't have env/experience of k3os)

Ok. One big issue which I see is at least you are using folder /var/vm because in BurmillaOS console is also running inside of container and view to it is different that which kernel see.
With quick look to code at least folder /var/lib/rancher/cache should be mounted to both so if you use SWAP_PATH=/var/lib/rancher/cache/swap it would fix at least that part.
On other why I'm not aware that there would be anything which would prevent you from using swap file.

(I don't know why above script works on v1.9.3, but :-)

Ok. Then it might be caused something else too.

I confirmed I can add a swap with SWAP_PATH=/var/lib/rancher/cache/swap.

But did it fix the issue?


ELEMENT_NODEs takes precedence over TEXT_NODEs while converting back to XML

Hi,
I am attempting to parse XLIFF 1.2 Sample File using this module.
This module is very much suitable for my requirements as it is lightweight as well as works both in client (browser) and server (node) environments.
I recently came across this issue while experimenting with it and wanted to report the same.
The details are as follows: -

Conversion from XML->JS is working as expected, but one thing, I noticed upon converting from JS->XML, the order/positions of TEXT_NODES inside <source> or <target> tags (both belongs to XLIFF) gets disturb; the nearby XLIFF Elements takes precedence over these TEXT_NODES.

Just wanted to check with you, is it the right behavior? OR I am missing some configuration.
For better clarity, please use the following links: -

The reported issue can be reproduced from this RunKit link.

The ORIGINAL and GENERATED XML outcomes can be checked via Diff Checker link.

Hope it helps you to understand the discussed problem in this use-case.
Kindly suggest a suitable fix for this issue.

Yes, this does appear to be a limitation of the library. The transformation is not order-preserving.
In theory, I suppose the library could be augmented to support this but I wonder if the magnitude of such a change might be outside the scope of a simple and easy to use library. Preserving order might not be very practical in terms of accurately reflecting modifications of the XML tree (how do you insert?) and so forth.
I would therefore consider this out of scope of the library. I suggest you find a SAX-style reader/writer library where the exact order and structure of XML is important to you.

Thanks for the reply.
In my use-case, order of these nodes is very important.
Hence as suggested by you would look for other/similar options.
Have a great day!!!


--disable-api-security does not disable API security

The change in behavior of the --disable-api-security introduced in v1.0.1 is unintuitive and breaks user scenarios.
In 1.0.1, if the user passes --disable-api-security siad requires the user to add an API password. This is confusing because this does not seem to honor the user's request to disable the security measures on the API port; it just applies different security requirements.
This requirement breaks scenarios like Sia via Docker. It is not possible for anything outside of the Docker container to communicate with the container's localhost interface. Even if the user tries to add an API password, it's klunky to script because siad expects the password to be entered by an interactive user rather than as part of a script.

good catch. In the meantime, you can listen on localhost and forward connections to the daemon. (I believe nginx can do this; there are probably more lightweight alternatives)

hmm.
This needs to be balanced against the fact that we have a lot of miners who will happily run the --disable-api-security command without properly securing their network and then lose all their coins. On one hand they kind of deserve it but on the other hand we need to protect them.
I'm not sure the best UX way to approach this, maybe @VoidingWarranties has some ideas? Goals:

protect miners/hosts with multi-machine setups
don't make life difficult for people using scripts and docker and whatever else

I think this falls into a similar bucket as people who don't write down their passwords. No matter how obvious you make the warnings, people are happy to click through and miss critical things. We need a better way to make people inherently secure.

API authentication was originally intended to protect against user-space malware on your local machine. As such, API auth uses HTTP basic authentication which sends passwords in plaintext. This is reasonably safe to do on the loopback interface, but is susceptible to MITM other interfaces.
Enabling authentication prevents against someone port scanning your machine and stealing your coins, but it also makes you vulnerable to a MITM attack if you make API requests from a compromised network.
An alternative solution to the docker annoyance is to allow the API password to be specified on the command line in a non-interactive way. Bitcoin allows you to do this with the -rpcauth flag which lets you specify the password's salt and hash. It wouldn't be difficult to do something similar here.

The --disable-api-security flag is a misnomer if we require API authentication to be enabled. It is not the best UX, but it seems to have reduced the number of vulnerable nodes.

The setup is secure by default, which is the right thing to do. The current behavior tries to make it "secure" no matter what the user wants, which I don't think is the right thing to do.
If the concern is that the users will just do whatever is easiest at the expense of security, they'll probably also just choose a weak API password. "a" as a password currently works.
Requiring Docker users to type a password on several different siac commands is a pretty big drag. Why punish Docker users who do the right thing in order to serve users who use non-default settings to create an insecure setup?

Makes sense.
We originally planned on making API auth enabled by default, but I don't think we ever did. We could enable it by default and have the --disable-api-security flag disable it. Or we could have the
--disable-api-security flag only enable non-localhost binding, and the users would still have to specify something like --authenticate-api false or maybe --disable-api-authentication. The latter seems like a safer option but maybe it's just more cumbersome. What do you think @mtlynch?
If you still want to enable API auth but don't want to type the password in siac interactively you can also specify the password like so --addr :<password>@localhost:9980.

If the concern is that the users will just do whatever is easiest at the expense of security, they'll probably also just choose a weak API password. "a" as a password currently works.

At least choosing the password 'a' is better than not having a password at all, it filters out the branch of criminal that didn't bother to do a basic dictionary attack on each autheticated api.
We only use basic http auth, which is a pretty terrible thing on the whole, if you can sniff a connection between a legit client and a legit server you can get the password to the server, there's no encryption. So, simply using authentication definitely doesn't count as secure, and security should still be considered 'disabled` if you are off localhost, even if you are using a password.
This change was inspired by a week of reports where numerous miners (correlating to the release of mining pools) were complaining about stolen funds. Most of these miners had exposed their apis with no authentication at all. Secure by default was not enough because they were definitely getting themsevles pwned, and I can't take a hard line when 10+ users are having large amounts of funds stolen. We needed to make a change in the more protective direction.
At some point you have to draw the line and say 'we refuse to protect users who are X dumb', but we need to make sure that those users don't form a significant part of the userbase.
I think the right thing to do is to make it easier for docker users to deal with the password. If typing it into the terminal is too much, perhaps an optional command line arg is more acceptable?
I will not be comfortable changing the name of --disable-api-security until we've got something superior to basic http auth, but that doesn't seem to be in the near future.

At least choosing the password 'a' is better than not having a password at all, it filters out the branch of criminal that didn't bother to do a basic dictionary attack on each autheticated api.

For the password to be a useful control, we have to assume that the most sophisticated attacker is someone who knows how to port scan and rob Sia wallets, but doesn't know how to perform a dictionary attack. Until the number of users running siad is in the tens of thousands, a single attacker performing dictionary attacks can feasibly rob every single Sia user with  a weak password. In other words, if we stop ten bad attackers by enforcing password requirements, but fail to stop one attacker capable of dictionary attacks, we didn't provide meaningful protection because that one attacker can compromise every Sia user.

This change was inspired by a week of reports where numerous miners (correlating to the release of mining pools) were complaining about stolen funds. Most of these miners had exposed their apis with no authentication at all. Secure by default was not enough because they were definitely getting themsevles pwned, and I can't take a hard line when 10+ users are having large amounts of funds stolen. We needed to make a change in the more protective direction.

Can you go into more detail about this attack? If you're pool mining, you don't need to run siad at all, so how could this happen? Even if they're running siad, what network are they on where there's just no firewall whatsoever? If I wanted to open up the siad API port to the Internet, I'd have to take several deliberate steps to do that, so I'm unclear how people are doing this accidentally.
FWIW, I just port scanned all the Sia hosts and none of them have port 9980 open to the Internet.
$  ./siac hostdb | grep -oP '[^\s]+:' | grep -oP [^:]+ > sia-hosts.txt
$ nmap -iL sia-hosts.txt -vvv -oA nmap-results -p 9980-9982 -Pn
$ grep 9982/open nmap-results.gnmap  | wc -l
65
$ grep 9980/open nmap-results.gnmap  | wc -l
0

Thanks for the responses @VoidingWarranties! My replies inline.

API authentication was originally intended to protect against user-space malware on your local machine. As such, API auth uses HTTP basic authentication which sends passwords in plaintext. This is reasonably safe to do on the loopback interface, but is susceptible to MITM on other network interfaces.

Sia should not attempt to defend against this scenario. If an attacker has code execution on the user's machine, it's game over. If there's an API password, they can sniff keystrokes or network traffic and very easily recover the password.

We originally planned on making API auth enabled by default, but I don't think we ever did. We could enable it by default and have the --disable-api-security flag disable it.

Requiring passwords by default seems unnecessary if the default is to listen on localhost. If we're listening on localhost, the password doesn't improve security because the attacker would have to already have code execution on the system if they're talking to localhost (assuming user hasn't set up strange port forwarding rules).

Or we could have the --disable-api-security flag only enable non-localhost binding, and the users would still have to specify something like --authenticate-api false or maybe --disable-api-authentication.

Do you have any particular scenarios in mind where we would want --authenticate-api false? For power users, it's, as you say, another cumbersome flag. For the confused user who's blindly adding --disable-api-security, presumably they'd just as blindly add --authenticate-api false, so we'd be in the same boat.

If you still want to enable API auth but don't want to type the password in siac interactively you can also specify the password like so --addr :<password>@localhost:9980.

That's nicer, but still not ideal. To me, the whole forced authentication feels like an imposition to power users that only superficially protects confused users who use insecure, non-default settings. So reducing the burden to power users is good, but I'm not clear on what we realistically win by burdening them in this way at all.

Ok, coming back to this in hopes of finding some closure.

FWIW, I just port scanned all the Sia hosts and none of them have port 9980 open to the Internet.

Most of the miners being compromised weren't running hosts. I'm guessing the attacker had some code to crawl the full list of nodes on the network (there's a nice file you keep locally with hundreds of entries) to scan ports and find vulnerable miners.

Sia should not attempt to defend against this scenario. If an attacker has code execution on the user's machine, it's game over. If there's an API password, they can sniff keystrokes or network traffic and very easily recover the password.

Userspace malware is not able to sniff keystrokes, nor is it able to sniff network traffic. You need root access to do those things. Malware has pretty limited abilities in userspace, we can use that to our advantage. They do have full access to the filesystem, which means they can do some nasty things with the consensus set, but that's a highly sophisticated attack at that point.

That's nicer, but still not ideal. To me, the whole forced authentication feels like an imposition to power users that only superficially protects confused users who use insecure, non-default settings. So reducing the burden to power users is good, but I'm not clear on what we realistically win by burdening them in this way at all.

If you are a power user, it's not that big of a burden to set up a password, and then specify the password as mentioned above. Normally I would side with you, --disable-api-security should be enough but we were still seeing users fall victim because they were just blindly adding that flag. Now that we've required both, I haven't seen any reports of vulnerabilities.

My favorite solution to all of this is to require the API to be served over localhost, and then have a guide for using standard tools (ssh, etc.) for talking to the API from another machine. If you are using an ssh server/client setup, or some equivalently secure setup, we don't have to worry about doing all of the security ourselves. Which is good, because security is hard and none of us are experts in this particular type of security.
I think we should consider the solution we have right now 'temporary', with the intention of blocking off exported API access in the near future.  Anyone who can sniff traffic can compromise you, and anyone able to guess the password can also compromise you.

Most of the miners being compromised weren't running hosts. I'm guessing the attacker had some code to crawl the full list of nodes on the network (there's a nice file you keep locally with hundreds of entries) to scan ports and find vulnerable miners.

Ah, gotcha. I assume you're talking about /gateway/nodes.json? I just dumped the IPs from this and scanned them. Of 1107 nodes, 22 of them are listening on 9980.
Do we know why these users are opening up port 9980 in the first place?
It feels like we're attempting to solve this problem at the wrong level. If Sia users are running into some problem and they're solving it (or at least they think they are) by opening up siad to the world, we should figure out what it is they can't do with the default, secure configuration and fix that.

Userspace malware is not able to sniff keystrokes, nor is it able to sniff network traffic. You need root access to do those things. Malware has pretty limited abilities in userspace, we can use that to our advantage. They do have full access to the filesystem, which means they can do some nasty things with the consensus set, but that's a highly sophisticated attack at that point.

I think defending against userspace malware is a waste of the Sia dev team's time. I think the right solution to contain malware is separate user contexts and containerization, not app-level protections. I can't think of any major software that tries to defend against userspace malware like this except for security software or OSes themselves.

My favorite solution to all of this is to require the API to be served over localhost, and then have a guide for using standard tools (ssh, etc.) for talking to the API from another machine. If you are using an ssh server/client setup, or some equivalently secure setup, we don't have to worry about doing all of the security ourselves. Which is good, because security is hard and none of us are experts in this particular type of security.

I think if the Sia dev team feels nervous about this aspect of security, the smartest thing to do is to model based on what other backend systems do. bitcoind, IIUC will allow you to do listen for RPCs without a password (and they also allow configuration via a config file, which is less painful than relying on interactive prompts).
But even outside of cryptocurrency, if we look at things like backend databases (MySQL, MongoDB, Redis), none of those will insist on only accepting commands over localhost. They also won't fight the user if the user elects to listen on all network interfaces.
One big reason is that it would get in the way of good software architecture. As people begin developing apps on top of Sia, it should be designed such that deployment is flexible and accommodating to service isolation. If we think about something like MegaUpload, but using Sia as a backend, an obvious architecture is probably something like:
            internet             |            backend
[ end user ] --> [ web server ] -|-> [ application server ] --> [ siad ]

Where each box is a separate machine or container. But if we're restrictive about the way that siad listens for RPCs, either the application server and siad have to be smushed into the same machine or the system has to be architected such that the app server talks to siad over SSH, which would be a huge pain.

Malware has pretty limited abilities in userspace, we can use that to our advantage. They do have full access to the filesystem, which means they can do some nasty things with the consensus set, but that's a highly sophisticated attack at that point.

I've been thinking about this a bit more because I really feel like trying to defend against malware running on the same host is a losing battle.

Do we know why these users are opening up port 9980 in the first place?

It feels like we're attempting to solve this problem at the wrong level.

Quite possibly because they are being instructed to by a mining pool. Either that, or they do not know any other way to connect a large number of mining computers to a single master siad instance.
I agree that resolving their use-cases is preferable to a heavy-handed ban on bad configurations. But I am not sure how broad the use cases are.

I think defending against userspace malware is a waste of the Sia dev team's time.

I can overwrite the siac binary with my own malicious binary that acts like siac but empties the victim's wallet as soon as it is unlocked.

Yes, there's not much we can do about that. Well, some operating systems I believe support binary-signing, which means you would not be able to do that. But it's not standard and we certainly aren't using it at the moment. And even if we were, the database file would need to be signed as well, because you can do similarly nefarious things by modifying the database file.
BUT, simply by preventing password files from ever being written to disk we can stop a broad range of script-kiddie malware, which I think is how most coins end up being stolen. It takes a much more sophisticated hacker to insert a replacement siac or to modify the database, so much so that the attacker has to be using outright sia-specific malware to execute any attacks.
General userspace malware is very common, and I consider it to be a huge gain that we protect against general userspace malware even if we are unable to protect against Sia-specific userspace malware.

I'm increasingly okay with removing the password requirement, but I still hesitate because I know that reports of stolen coins dramatically decreased as soon as we added the password requirement. We have pretty strong evidence that the password requirement has actively saved users money, to the tune of millions of siacoins.
All of the vulnerable users are already using passwords now, so dropping it may mean that they will continue to use passwords. I think we should add a warning in red in the output though - "This setup puts your coins at risk of being stolen - adding a password with '-p mypassword' is strongly recommended." - to catch any users who are ignorantly exposing their API to the public.
We should probably have the warning in red even with the password, given that it's basic HTTP auth and no good against anyone who can sniff the traffic.

Quite possibly because they are being instructed to by a mining pool.

I'm not sure I follow. Why would a mining pool instruct anyone to do that? My understanding is that anyone pool mining doesn't need to run siad at all.

General userspace malware is very common, and I consider it to be a huge gain that we protect against general userspace malware even if we are unable to protect against Sia-specific userspace malware.

I'm confused by this. Isn't Sia-specific malware all we're talking about? Any malware that puts Sia at risk has to be Sia-aware. There isn't any generic malware that can figure out what Sia is without having ever been programmed specifically to attack Sia.

I know that reports of stolen coins dramatically decreased as soon as we added the password requirement. We have pretty strong evidence that the password requirement has actively saved users money, to the tune of millions of siacoins.

Are any of these reports public? I tried searching the sia.tech forums and couldn't find reports of Sia users having coins stolen.

I'm not sure I follow. Why would a mining pool instruct anyone to do that? My understanding is that anyone pool mining doesn't need to run siad at all.

I am not sure either. My assumption was that some how-to guide somewhere had some bad advice, just based on the types of reports we were getting. I should have investigated further.

I'm confused by this. Isn't Sia-specific malware all we're talking about? Any malware that puts Sia at risk has to be Sia-aware. There isn't any generic malware that can figure out what Sia is without having ever been programmed specifically to attack Sia.

There's a lot of malware that has generic tools to sweep up anything that looks like a wallet file or a password file. Considering the sia wallet is called 'wallet.json', it's not too much of a stretch to see it getting picked up. The wallet used to be called 'wallet.dat', which is the name of the bitcoin file, which is much more likely to get swept up.
I guess I am not an expert on it, but my understanding has always been that most malware is run by people other than the writers of the malware.
Other than some active network-based attacks (which did seem to be Sia-specific), I don't think I've seen any reports congruent with wallet files being stolen. I do think that this is in-part due to the fact that we refuse to save the user password to disk. Hard to be certain though, as we don't have any sort of A/B testing in our releases.

Are any of these reports public? I tried searching the sia.tech forums and couldn't find reports of Sia users having coins stolen.

Some were publicly discussed in the slack, many in the #mining channel. Several were just private messages directly to me. I don't think that any made it all the way to the forum.

I think you might be confusing this discussion with #1403. We're not talking about saving a password to file here, but rather requiring a password on the siad API.
@VoidingWarranties said this:

API authentication was originally intended to protect against user-space malware on your local machine. As such, API auth uses HTTP basic authentication which sends passwords in plaintext. This is reasonably safe to do on the loopback interface, but is susceptible to MITM on other network interfaces.

I'm saying that for malware to do this, it has to be programmed specifically to target Sia. If the attacker is going to the trouble of writing Sia-aware malware, they might as well replace the binary with an evil siac binary that drains the wallet as soon as the user enters their wallet password.
So with this in mind, I don't think an API password for siad offers any protection against userspace malware.
I understand what you're saying in that it offers some degree of protection to users who expose 9980 to the public Internet. If you require a password to listen outside localhost, at least now the attacker has to do the work of brute forcing the password, but I see this as pretty trivial for a Sia-focused attacker. So it effectively becomes a speed bump to attackers, but a real problem for legitimate users who want a painless way to deploy siad in things like Docker containers or isolated VMs.

Closing per @DavidVorick's comment in #1387.
I've tested a workaround that uses socat to port forward to 9980:
socat tcp-listen:8000,reuseaddr,fork tcp:localhost:9980 & ./siad

siad is happy because it's only listening on localhost:9980, but we can still access it from 8000 as a non-localhost port.

Thanks for posting a workaround. I hope that we can revisit this at some point and arrive at a solution that doesn't require a workaround.

New suggestion.
why not use Token ? 1024 bytes token. with user key.

SIAD request to SIA NET : upload access token
SIA BLOCKCHAIN NETWORK : give temporary token
SIAD -> pass to app (iOS,Android)
APP : upload to SIA BLOCKCHAIN NETWORK

download same.
we can make the new fantasy world.


'eosiolib/currency.hpp' file not found

all current github examples using the following signature:
void onTransfer(const eosio::currency::transfer& transfer);
Looks like currency.hpp is removed in eosio.cdt, then how to compile?

You can use -I=/pathto/eos/ parametr
for example eosio-cpp ../src/pvp.cpp -o ../build/pvp/pvp.wasm -I=/mnt/d/EOS/eos/contracts


Loading uncompiled MIB files.

I'm stumped on what I would think would be a simple task. The sample script "SNMP MIB browser" gives every indication that compiler.addMibCompiler(mibBuilder, sources=['/usr/share/snmp/mibs']) adds a MIB compiler to load uncompiled MIBs and that mibBuilder.addMibSources(builder.DirMibSource('/opt/pysnmp_mibs')) is to add a source containing pre-compiled MIBs.
#!/usr/bin/env python
from pysnmp import debug
from pysnmp.smi import builder, view, compiler, rfc1902, error

def main():
#    debug.setLogger(debug.Debug('all'))
    mibBuilder = builder.MibBuilder()
    compiler.addMibCompiler(mibBuilder, sources = ['file:///home/user/python-udp/MIBs', ])
    mibBuilder.loadModules('SNMPv2-SMI')
    mibBuilder.loadModules('SNMPv2-TC')
    mibBuilder.loadModules('INET-ADDRESS-MIB')
    mibBuilder.loadModules('IANAifType-MIB')
    mibViewController = view.MibViewController(mibBuilder)
    modName = mibViewController.getFirstModuleName()
    while 1:
        if modName: print(modName)
        try:
            modName = mibViewController.getNextModuleName(modName)
        except error.SmiError:
            break
    return

if __name__ == '__main__':
    main()

The loadModules() all work until the one to load IANAifType-MIB which is the first one which isn't included in pysnmp itself. At no point does it give any indication that it's looking in the directory I specified. I've tried numerous variations on the theme but cannot find anything which works. Is there a concise code snippet for loading uncompiled MIBs? Thanks in advance.
Traceback (most recent call last):
  File "./loadmib.py", line 24, in <module>
    main()
  File "./loadmib.py", line 12, in main
    mibBuilder.loadModules('IANAifType-MIB')
  File "/home/user/python-udp/venv/lib/python2.7/site-packages/pysnmp/smi/builder.py", line 388, in loadModules
    raise error.MibNotFoundError('%s compilation error(s): %s' % (modName, errs))
pysnmp.smi.error.MibNotFoundError: IANAifType-MIB compilation error(s): missingcaused by <class 'pysnmp.smi.error.MibNotFoundError'>: MIB file "IANAifType-MIB.py[co]" not found in search path (DirMibSource('/home/user/python-udp/venv/lib/python2.7/site-packages/pysnmp/smi/mibs'), DirMibSource('/home/user/python-udp/venv/lib/python2.7/site-packages/pysnmp/smi/mibs/instances'), DirMibSource('pysnmp_mibs'), DirMibSource('/home/user/.pysnmp/mibs'))caused by <type 'exceptions.IOError'>: [Errno 2] No suitable module found: 'IANAifType-MIB'

Just a wish: it would be very helpful if the exception contained the actual error reason, e.g. "Couldn't find dependency WHATEVER-MIB for SOME-MIB" or "Error in SOME-MIB", etc. The currently reported "file not found" is extremely confusing.
I'm facing a similar problem now with pysnmp 7.1.8 and this simple program:
from pysnmp.hlapi import *
from pysnmp.smi import builder, view, compiler, rfc1902
from pysnmp import debug
debug.set_logger(debug.Debug('all'))
mibBuilder = builder.MibBuilder()
mibViewController = view.MibViewController(mibBuilder)
test = ObjectIdentity('ATEN-PE-CFG', 'outlet23Status', 0).add_asn1_mib_source('/opt/conserv-tools-venv/usr/share/snmp/mibs', '/usr/share/snmp/mibs',)
test.resolve_with_mib(mibViewController)
test.getOid()

I used strace and found out that it DOES succesfully open the ATEN-PE-CFG.txt file and reads it, but then it still reports this:
2024-10-31 20:10:48,355 pysnmp: loadModules: calling MIB compiler for ATEN-PE-CFG
Traceback (most recent call last):
  File "/opt/conserv-tools-venv/lib/python3.12/site-packages/pysnmp/smi/builder.py", line 398, in load_modules
    self.load_module(modName, **userCtx)
  File "/opt/conserv-tools-venv/lib/python3.12/site-packages/pysnmp/smi/builder.py", line 374, in load_module
    raise error.MibNotFoundError(
pysnmp.smi.error.MibNotFoundError: MIB file "ATEN-PE-CFG.py[co]" not found in search path (DirMibSource('/opt/conserv-tools-venv/lib/python3.12/site-packages/pysnmp/smi/mibs'), DirMibSource('/opt/conserv-tools-venv/lib/python3.12/site-packages/pysnmp/smi/mibs/instances'), DirMibSource('pysnmp_mibs'), DirMibSource('/h
ome/amelkin/.pysnmp/mibs'), DirMibSource('/home/amelkin/.pysnmp/mibs'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/amelkin/conserv-tools/./snmpv3.py", line 47, in <module>
    test.resolve_with_mib(mibViewController)
  File "/opt/conserv-tools-venv/lib/python3.12/site-packages/pysnmp/smi/rfc1902.py", line 542, in resolve_with_mib
    (mibNode,) = mibViewController.mibBuilder.import_symbols(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conserv-tools-venv/lib/python3.12/site-packages/pysnmp/smi/builder.py", line 454, in import_symbols
    self.load_modules(modName, **userCtx)
  File "/opt/conserv-tools-venv/lib/python3.12/site-packages/pysnmp/smi/builder.py", line 416, in load_modules
    raise error.MibNotFoundError(
pysnmp.smi.error.MibNotFoundError: ATEN-PE-CFG compilation error(s): missing; missing caused by <class 'pysnmp.smi.error.MibNotFoundError'>: MIB file "ATEN-PE-CFG.py[co]" not found in search path (DirMibSource('/opt/conserv-tools-venv/lib/python3.12/site-packages/pysnmp/smi/mibs'), DirMibSource('/opt/conserv-tools-ve
nv/lib/python3.12/site-packages/pysnmp/smi/mibs/instances'), DirMibSource('pysnmp_mibs'), DirMibSource('/home/amelkin/.pysnmp/mibs'), DirMibSource('/home/amelkin/.pysnmp/mibs'))

I'm currently quite stuck. Any help would be appreciated. Thanks.
Full log is attached: pysnmp.log
P.S. Do you know why latest Ubuntu 24.04 LTS still has pysnmp 4.4.12 ? Is there any place where documentation for that extremely outdated version can still be found? I failed to google it.

Oh, almost as soon as posted the comment, I have found out the reason it was failing. The MIB I was loading was demanding RFC-1212 and RFC-1215 MIBs that weren't installed by Ubuntu's snmp-mibs-downloader and weren't provided by Aten along with their ATEN-PE-CFG MIB.
Nonetheless, I was able to find that out only using strace because pysnmp or pysmi didn't give a slightest hint regarding what was wrong and kept reporting 'file not found' about the main MIB file that clearly was present.
Clearer error messages could save me a day of debuggng.

@f1r35pirit this repo is abandoned, #429, so if you wish the error message to be clearer, this isn't the place to report.

i was facing the similar issue that my private mib is not compile.
though inside that mib there is dependency
error from <pysmi.codegen.pysnmp.PySnmpCodeGen object at 0x7aa96ec9d900>: no symbol "Unsigned32" in module "SNMPv2-SMI" at MIB MY-MIB
how do i resolve the issue of "Unsigned32"?

@kdarjoshi That indicates you are using very wrong versions of PySNMP and PySMI. Please read #429 to learn where to get started.


Increment Version

@JuliaRegistrator register() ACTRModels v0.4.1

Error while trying to register: "Tag with name v0.4.0 already exists and points to a different commit"

@JuliaRegistrator register() ACTRModels v0.4.1

Registration pull request created: JuliaRegistries/General/22590
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.4.1 -m "<description of version>" ba0b6eb757eebf389ecec69fcac2ea73c30c9c68
git push origin v0.4.1


F-Droid release

The eduroamCAT tool is available on F-Droid. This tool is not yet. Are there plans to do so?

Definitely. It was on my agenda for 2024 to do after the release of the new Apps, I just didn't get to it yet ;-)

Awesome! Thanks! I personally run a custom ROM to avoid google and to maximize the amount of free software in my life, so this is great news. Nice that this is on the planning, even though it is a thing only a few users will care about. Thanks again!

Any news on this?

I opened a new issue here: https://gitlab.com/fdroid/rfp/-/issues/2890
I’ve also done some initial work on packaging here: https://gitlab.com/tibequadorian/fdroiddata/-/tree/app.eduroam.geteduroam
For inclusion, this repository should include Fastlane files: https://gitlab.com/-/snippets/1895688
While packaging, I found the versioning of this app confusing. There are many tagged snapshots without bumping the version number and going by the date for the latest update on the play store, the current stable release seems to be 2.0.1-androidsnapshot-1926? Proper versions would also streamline the inclusion on F-Droid.

A Github release would be great too. Thus people using Obtainium could directly install from there.

I'd like to second this suggestion.


Update unicodeset link

The syntax changed and the links broke.
r? @centril

Also wondering if I should update the RFC to use the term uncommon_codepoints (the RFC says less_used_codepoints but it left that up as an unresolved question and the tracking issue has settled on uncommon_codepoints), but my understanding is that we typically don't update RFCs to reflect decisions made later? (@nikomatsakis ?)

but my understanding is that we typically don't update RFCs to reflect decisions made later? (@nikomatsakis ?)

We don't require it, but I personally wouldn't mind accepting & reviewing such fixes and clarifications.


Fuzz: Check for missing channels in ModularImageToDecodedRect

gi.channels may not contain all the channels we expect so we need to
check for them before accessing the array.

Replaced by #379


Ed25519 EVP Support

I looks like the Ed25519 EVP is now supported in this commit
My question is this commit is merged in which openssl version?

Note that Ed25519 was already supported, it's just the Ed25519ctx, Ed25519ph variants that are now also supported.
That commit is only part of the master branch, and is not in any release.
Ed25519 is supported since 1.1.1.

Note also that ED25519 and X25519 (as 448) are also not currently approved for FIPS mode.

My understanding is that X25519 is permitted whereas Ed25519 isn't allowed.

Closing since this is marked as resolved.


Improve component bundling and reduce total downloaded files

Description
Related: #954
Prior work: #1345
Issue in the official Webpack implementation: https://github.com/facebook/react/issues/22314
This brings some updates from the RSC Vite plugin with these goodies:

Emit chunks for only the client components used in the project
Only create import-glob references for the client for components that are actually used
Bundle client components that are used together in the same chunk (only one "client boundary") across 3p libs and user app. Examples: Seo* or Cart*.

It reduces the total number of JS files downloaded to the browser from 67 to 34 🎉 . The remaining JS files related to client components are either legit client components that are imported directly in the server, or chunks that contain code used by multiple components (e.g. LocalizationContext.client or SpinnerIcon.client).
There are some limitations, though: the connection between client components cannot be discovered if they are re-exported from non-facade files, such as re-exporting a client component from a shared component. It works when re-exporting from facade files like index.ts any number of times (a facade file has no logic other than re-exports).
Also, it only works with named exports that are not renamed in facade files (discovering connections with default exports or renamed exports requires more work, but possible).
To see the changes more clearly, check this commit in the RSC Plugin PR.

Before submitting the PR, please make sure you do the following:

[x] Read the Contributing Guidelines
[x] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123)
[ ] Update docs in this repository according to your change
[x] Run yarn changeset add if this PR cause a version bump based on Keep a Changelog and adheres to Semantic Versioning

I don't entirely understand what's going on in this PR. I pulled down the changes, and tested it out. On the home page total requests went from 72 to 48 🥳

Interesting. So you are bring back the bundler config ... but more accurately, made it available at the vite server level. Then react server dom vite plugin goes into the module graph to look for a entry route to the module?

My bad, should have explained the overall approach. It's something like this:

Augment Vite's module graph with each module's import/export information (exported variables, imported variables and paths, etc). This information does not exist in the raw module graph provided by Vite.
Use the module graph to trace client component imports: traverse the graph upwards from each client component to explore every importer (recursively to make it work with multiple index.ts file re-exports).
If a client component turns out to be imported only by another client component, then we can bundle them together (this happens with all our Seo* components).
If a client component is imported by a server component, then we need to emit a separate chunk for it (becomes a "client boundary").


alt: clean JS, better implementation using classes

Review: The code here can be shortened by creating classes and defining styles in the stylesheet.
https://github.com/dheerajdlalwani/back-to-work/blob/fc5c9820ee350dc1ab69f9c9aff77465bc33f979/src/popup/popup.js#L7-L18
It can be shortened more using a default, which I assume is Chill. Have the HTML, state & styles initially for that.
Have a class for Work, and toggle that whenever necessary.
 (mode === "Work") ? modeBtn.classList.add("work") : modeBtn.classList.remove("work");

Now CSS for work would be something like:
.work {
  content: "work";
  background-color: #f95050;
  color: #ffffff;
}

Reason to do that?
Keeps the JS clean and and implementation short. No need for item.style.backgroundColor etc.

Ahh, this suggestion helped me clean up my code quite a bit!
Thank you so much for raising this issue 😄


TESSDATA_PREFIX Environment Variable set but it still cannot find it

Error
Error opening data file tessdata/eng.traineddata
Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory.
Failed loading language 'eng'
Tesseract couldn't load any languages!

But I do have the environment variable set to the tessdata folder with eng.traineddaata.

It only works if I call it from my project root directory where tessdata folder is published into.
Works
PS D:\Development\Blitz\Blitz\bin\Release\net7.0\win-x86\publish> blitz scan

Does Not Work
PS C:\Users\mdabr> blitz scan
Error opening data file tessdata/eng.traineddata
Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory.
Failed loading language 'eng'
Tesseract couldn't load any languages!
Unhandled exception. Tesseract.TesseractException: Failed to initialise tesseract engine.. See https://github.com/charlesw/tesseract/wiki/Error-1 for details.
   at Tesseract.TesseractEngine.Initialise(String datapath, String language, EngineMode engineMode, IEnumerable`1 configFiles, IDictionary`2 initialValues, Boolean setOnlyNonDebugVariables)
   at Tesseract.TesseractEngine..ctor(String datapath, String language, EngineMode engineMode, IEnumerable`1 configFiles, IDictionary`2 initialOptions, Boolean setOnlyNonDebugVariables)
   at Tesseract.TesseractEngine..ctor(String datapath, String language, EngineMode engineMode)
   at Blitz.Program.RunScanMoronCommand(ScanMoronOptions opts) in D:\Development\Blitz\Blitz\Program.cs:line 156
   at Blitz.Program.<>c.<Main>b__1_3(ScanMoronOptions opts) in D:\Development\Blitz\Blitz\Program.cs:line 27
   at CommandLine.ParserResultExtensions.MapResult[T1,T2,T3,T4,T5,TResult](ParserResult`1 result, Func`2 parsedFunc1, Func`2 parsedFunc2, Func`2 parsedFunc3, Func`2 parsedFunc4, Func`2 parsedFunc5, Func`2 notParsedFunc)
   at Blitz.Program.Main(String[] args) in D:\Development\Blitz\Blitz\Program.cs:line 16

Workaround
I will not close this issue since the error persists. However, for anyone experiencing similar problems, I have found a workaround by programmatically creating the tessdata directory and downloading eng.traineddata to a known location in the user's file system on app initialisation.
Get a known path (e.g. AppData)
Create the tessdata directory there.
private const string AppDataFolderName = "YourAppName";
private const string TessdataFolderName = "tessdata";

/// <inheritdoc/>
public string GetAppDataFolderPath()
{
    var appDataPath = Environment.GetFolderPath(Environment.SpecialFolder.ApplicationData);
    return Path.Combine(appDataPath, AppDataFolderName);
}

/// <inheritdoc/>
public string GetTessdataFolderPath()
{
    return Path.Combine(GetAppDataFolderPath(), TessdataFolderName);
}

Download tessdata/*.traineddata to that path
Make sure to check the directory exists there before downloading.
/// <summary>
/// Downloads the Tesseract English language model to the tessdata folder.
/// </summary>
/// <param name="tessdataFolderPath">The path to the tessdata folder.</param>
private static async Task DownloadTrainedData(string tessdataFolderPath)
{
    const string tessdataEngFileName = "eng.traineddata";
    const string tessdataEngUrl = "https://github.com/tesseract-ocr/tessdata_fast/raw/main/eng.traineddata";

    using var client = new HttpClient();

    await using var stream = await client.GetStreamAsync(tessdataEngUrl);
    await using var fs = new FileStream(Path.Combine(tessdataFolderPath, tessdataEngFileName),
        FileMode.OpenOrCreate);

    await stream.CopyToAsync(fs);
}

Initialize the engine using your defined path
using var engine = new TesseractEngine(tessdataFolderPath, "eng", EngineMode.Default);


How do I have a Blog with Bludit 2.0

Hello Developers,
I have Bludit 2.0 Beta 3 installed and found that only frontend pages are displayed. But how can I, for example, create a page called /blog and all blogposts end up on this page.
So far I have not understood it yet.
Maybe someone can help me.
Thanks in advance.

On Bludit 2, pages and posts are the same

Hi,
just install Bludit v2.0, and create a new page, that it.
By default the content is ordered by date, so is a Blog.
If you need to create a Static page such as "About", "Contact", create a new page and on the menu Advanced->Status->Select Static.
Regards

Check the new beta 4, https://github.com/dignajar/bludit/releases/tag/2.0-beta4

Just updated but the problem remains :/

I close the ticket. Please take a look at the new beta version to check possibles fixes for the issue, https://github.com/dignajar/bludit/releases/tag/2.0-beta7
Feel free to open a new ticket if the problem persists with the latest version.


Insiders: Cannot signal tasks

VSCode Version: Version: 1.47.0-insider
Commit: 0913b1aa43191d8af0ccb4a133d9a8d7c1a81d69
Date: 2020-06-23T09:38:28.751Z (13 hrs ago)

Checked again and it seems to repro even in 1.46

OS Version: macOS 10.15.4

Steps to Reproduce:

Create a new extension using yo code, named test
Add the following code to the activate function body, you might need to change the command id:

	let disposable = vscode.commands.registerCommand('test.helloWorld', async () => {
		let task = new vscode.Task(
			{type: "taskbug"},
			vscode.TaskScope.Workspace,
			'echo',
			'taskbug',
			new vscode.ShellExecution('trap "exit 8;" INT; trap "exit 9;" TERM; sleep 1000000'));

		const echoTaskExecution = await vscode.tasks.executeTask(task);

		vscode.tasks.onDidStartTaskProcess(e => {
			if (e.execution === echoTaskExecution || e.execution.task === task) {
				vscode.window.showInformationMessage(
					`Detected that my task started with pid ${e.processId}`
				);
				setTimeout(() => {
					vscode.window.showInformationMessage(
						`Sending sigint`
					);
					process.kill(e.processId, 'SIGINT');
				}, 1000);

				setTimeout(() => {
					vscode.window.showInformationMessage(
						`Sending sigterm`
					);
					process.kill(e.processId, 'SIGTERM');
				}, 3000);

				setTimeout(() => {
					vscode.window.showInformationMessage(
						`Using task API to end`
					);
					e.execution.terminate();
				}, 10000);
			}
		});

		vscode.tasks.onDidEndTaskProcess(e => {
			if (e.execution === echoTaskExecution || e.execution.task === task) {
				vscode.window.showInformationMessage(
					`Detected that my task ended with exit code ${e.exitCode}`
				);
			}
		});
	});

	context.subscriptions.push(disposable);

Trigger the command.

Watch the above notifications to see what happens. Sending SIGINT and SIGTERM do nothing - the only way to end the task is to call execution.terminate(); or have the user CNTRL+C in the terminal, but in my case I want to be able to programmatically signal my task since it registers specific signal handlers.

Nevermind, looks like this is an issue with the shell - if I change it to use Python instead it works.


Using Cloud Firestore instead of EF & SQL for Identity.

For user authentication and authorization I want to use ASP.NET Core Identity. All default setups are with entity framework using SQL.
I want to use Firestore as my project is using it already.
Is there any helper package just to replace EF and keeping the Identity implementation similar with Firestore? Something that replaces AddEntityFrameworkStores<AppDbContext>() for Firestore.
Or will I have to do it manually sending/getting data to Firestore on my own as we do for everything else?
services
    .AddIdentity<IdentityUser, IdentityRole>()
    .AddEntityFrameworkStores<AppDbContext>()  // <<= Any replacement for Firestore ?
    .AddDefaultTokenProviders();

What you are looking for is a Firestore based Custom storage provider for ASP.NET Core Identity and I'm pretty sure we don't maitain anything like that (@jskeet for final confirmation). But I believe you could write your own following the instructions in the link above.

Indeed, I'm afraid we don't have that :(

I'll be closing this issue now, since I believe we have answer your question. Feel free to leave a comment here if you believe otherwise.
Also, for Google APIs client libraries (including Google.Cloud.Firestore.V1) related issues, it is better if you post them at https://github.com/googleapis/google-cloud-dotnet.
Post in this repo we are in if you find issues with the samples themselves.


Bookmarks should lead to the item in the context of the collection they were bookmarked from

Currently if I bookmark the same object from different collections, my bookmark link says I have 2 bookmarks, but when I go to my bookmarks page it only says / shows 1. I believe repository retrieval deduplicates based on the manifest url. This is probably desirable in the general search results screen, but maybe not in bookmarks? If what we care about is exhibit-specific metadata?


[MINOR][DOCS] Fix rst link in Python API docs for .sql()

What changes were proposed in this pull request?
This PR fixes the rst markup for a link in the documentation for pyspark.sql.SparkSession.sql and pyspark.pandas.sql.
Why are the changes needed?
The current markup is incorrect.
Technically, though the markup in this PR is correct, the link target is incorrect. We should be linking to page relative to the site root, rather than hardcoding a link to /latest/. However, I could not figure out how to do that in rst, and building the API docs takes a really long time, and I could not make it build incrementally.
Does this PR introduce any user-facing change?
Yes, the markup goes from looking like this:

To looking like this:

How was this patch tested?
I built the docs as follows:
SKIP_SCALADOC=1 SKIP_RDOC=1 SKIP_SQLDOC=1 bundle exec jekyll serve

And reviewed the output in my browser.
Was this patch authored or co-authored using generative AI tooling?
No.

cc @MaxGekk since git blame suggests you wrote the docs I am editing in this PR.

Build: https://github.com/nchammas/spark/actions/runs/7325659058/job/19950500574 (for my own ref)

Merged to master.


I need some examples for knowing how to use your tool

Dear Sir,
Please publish some examples for training us.
I installed your tool, but no document.
Best regards.

The samples are part of the functions documentation, if you use get-help  -detailed you will get examples


Update configure-your-spam-filter-policies.md

On the Actions page that appears, configure the following settings:

Message actions > Remove "High confidence phishing" checkmark against "Move message to Junk Email folder:" action based on following MC237394 and below documentation. https://learn.microsoft.com/en-us/microsoft-365/security/office-365-security/secure-by-default?view=o365-worldwide We have deprecated the Move message to Junk Email folder action for a High confidence phishing email verdict in EOP anti-spam policies. Anti-spam policies that use this action for high confidence phishing messages will be converted to Quarantine message. The Redirect message to email address action for high confidence phishing messages is unaffected.

Learn Build status updates of commit 3672788:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/office-365-security/configure-your-spam-filter-policies.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Learn Build status updates of commit 83975ee:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-365/security/office-365-security/configure-your-spam-filter-policies.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


Dimaio/bb dep omega

This code adds a backbone-dependent omega term to tmol.  It matches the R3 implementation with 2 exceptions:

residues w/ undefined phi/psi are mapped to basic (non-bbdep) variant instead of setting undefined torsion to 0
preproline tables are not always used in R3 when they should be

The code replaces the rama and omega score terms with a single term, 'backbone_torsion,' that computes both in a single kernel.

Question purely for documentation purposes -- what are the places in R3 where prepro isn't used where it should be? (Maybe this could inspire some improvements in R3.)


Suricata Plugin

Elasticsearch version:7.3.1
Moloch version:2.0.0
OS name and version:Ubuntu 18.04
How was Moloch built/installed: (rpm, deb, easybutton, ...) deb
Provide logs, stack traces and steps to reproduce:
(1) enabled and configured suricata
(2) navigated to SPIView
(3) loaded all under suricata
(4) click on one of the signatures (i.e. generate a filter to pivot)
(5) filter output = "suricata.signature == SURICATA STREAM excessive retransmissions"
(5a) results in an error
(5b) to remedy/fix you have to add quotations after the == (example: suricata.signature == "SURICATA STREAM excessive retransmissions")
Low Priority Issue

Duplicate of #1146 already fixed in 2.0.1, please upgrade


Improved walking Hammerdin in Trav (clears it more consistently w/o getting stuck and loots everything)

The current scripting for Hammerdins without Enigma results in a large variance in how much of the council are killed, and it very frequently gets stuck looting below or above the window.
One big problem with the current script is that it would start by running all the way inside the building.  These changes now begin Hammers outside, at the entrance, so that is no longer an issue. The mobs will generally come out to meet you. Afterwards, it goes into the building to clean up anyone who stayed inside.
The other big problem with the current script is with looting. The wall separating the interior of the building and the outside patio area caused the bot to get stuck looting on the wall - frequently skipping half or even all of the loot. My changes now check both the inside and outside for loot in separate steps when Teleport is not available.
I made sure to split these changes out in such a way that only a non-Teleporting character will be affected.

Often getting stuck outside trav. Specifically at this step [2022-01-24 18:28:47,421] DEBUG      Traverse: [230]

This PR has been working great for my under geared walking hammerdin.
Avg run 1:46 trav only.
Very thorough drops and consistently walking through the door and killing council.

*Huge improvement from default walk.
The walking version will never be a perfect science, but this handles all the problems with blocking and looting very well. 100% this should be part of the master/default. Instead of derp-ing on a block, it continues trying to fight/move and even loot to be safe. Very quick runs too.
Basic gear, no enigma, hoto/shako/hoz/cta but it's pretty-much overkill.
70-80 second runs seem to be the avg. (My last 20 had 74 secs avg)
Great work, ty!

@aeon0 This PR should be good to go! I have a bunch of people currently running it. Here's a screenshot from today using these changes (note: I have a pretty selective pickit, so the number of items would be higher for others in the same number of runs):

Apparently, my changes did affect Enigma hammerdins, so I'm gonna fix that and re-submit


Infer data type from dataset

Infer data_type (func or anat or both) from dataset rather than use command line parameter.

The data_type is actually inferred from the dataset, the --data_type option is there to restrict the analysis to one of those if the user does not want to process all. In practice, if you have a dataset with both anatomical and functional data, you'd use --data_type to run only the anatomical or only the functional.
However, this issue here comes from a bug in the group level. It breaks in several use cases. This issue will be closed when those bugs are fixed.


Newton cradle

About the PR

Why / Balance

Technical details

Media

https://github.com/space-wizards/space-station-14/assets/124214523/c360a4a3-fd36-41b4-9193-bb7ef1d44e77

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

Changelog

no cl no fun

Needs the sound.

ahhh

added

personally, i would like the side pendulums to go a bit higher on-sprite
otherwise, 👍

With the sound, might want to add a way to toggle it to a static state.

Second this, would be a good if there was a way to toggle it on and off, otherwise I have a feeling it's going to get yeeted into disposals pretty quick


⚠ Bofrost.at has degraded performance

In f313329, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2876 ms

Resolved: Bofrost.at performance has improved in 05b9577.


Error exporting existing ACI configuration to YAML (identity object)

This is autogenerated. Please review and update as needed.
Describe the bug
Command Name
az container export
Errors:
The command failed with an unexpected error. Here is the traceback:
('cannot represent an object', <azure.mgmt.resource.resources.v2021_04_01.models._models_py3.Identity object at 0x104c130d0>)
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/container/custom.py", line 633, in container_export
    yaml.safe_dump(resource, f, default_flow_style=False)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/yaml/__init__.py", line 269, in safe_dump
    return dump_all([data], stream, Dumper=SafeDumper, **kwds)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/yaml/__init__.py", line 241, in dump_all
    dumper.represent(data)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/yaml/representer.py", line 58, in represent_data
    node = self.yaml_representers[None](self, data)
  File "/opt/homebrew/Cellar/azure-cli/2.42.0/libexec/lib/python3.10/site-packages/yaml/representer.py", line 231, in represent_undefined
    raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', <azure.mgmt.resource.resources.v2021_04_01.models._models_py3.Identity object at 0x104c130d0>)

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az container export --output {} --resource-group {} --name {} --file {}

Expected Behavior
Environment Summary
macOS-13.0.1-arm64-arm-64bit, Darwin 22.1.0
Python 3.10.8
Installer: HOMEBREW

azure-cli 2.42.0

Additional Context

route to CXP team


[$250] [LOW] [Splits] IOU - Header of IOU transaction thread is not grayed out when created offline

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.58-4
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail: https://expensify.testrail.io/index.php?/tests/view/4466053
Logs: https://stackoverflow.com/c/expensify/questions/4856
Issue reported by: Applause-Internal team
Action Performed:

Go to staging.new.expensify.com
Go offline.
Request money from a user with no previous chat.
Go to 1:1 DM, IOU report and transaction thread.
Note the header of the three reports.

Expected Result:
The header of 1:1 DM, IOU report and transaction thread should be all grayed out.
Actual Result:
The header of 1:1 DM and IOU report is grayed out, but not for transaction thread.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~017d9117dc1c8dc5c7
Upwork Job ID:<PHONE_NUMBER>421372928
Last Price Increase: 2024-04-15

Proposal
Please re-state the problem that we are trying to solve in this issue.
The transaction thread header is not grayed out when created while offline which is inconsistent with the chat and IOU/expense report header.
What is the root cause of that problem?
When we create a new request while offline, we will optimistically create the chat and IOU report (if new) and add a pending fields.
https://github.com/Expensify/App/blob/c848cb6bf29b600eecb8b338222fffe8a36d0329/src/libs/actions/IOU.ts#L516
https://github.com/Expensify/App/blob/c848cb6bf29b600eecb8b338222fffe8a36d0329/src/libs/actions/IOU.ts#L529-L531
The pending fields will be used to grayed out the whole report screen, including the header.
However, we never add a pending fields for the optimistic transaction thread.
https://github.com/Expensify/App/blob/c848cb6bf29b600eecb8b338222fffe8a36d0329/src/libs/actions/IOU.ts#L573-L575
Thus, both the header and the composer are not grayed out.
What changes do you think we should make in order to solve the problem?

Add a pending fields the same as chat/IOU report.
value: {
    ...transactionThreadReport,
    pendingFields: {createChat: CONST.RED_BRICK_ROAD_PENDING_ACTION.ADD},
},

Then, clear it when fails just like we did for chat/IOU report too. (we already clear it when success)

Seeing as this is IOU related I'm assigning to vip-split

Let's fix!

Waiting on proposals

Assigning @bernhardoj

PR is ready
cc: @hoangzinh

Payment summary:
C $250 @bernhardoj (Upwork)
C+ $250 @hoangzinh (Upwork)

Done! Thanks


Use Vuex with persistent state in v2

State management: https://vuex.vuejs.org/en/
Keep state: https://github.com/robinvdvleuten/vuex-persistedstate
Native storage: https://github.com/TheCocoaProject/cordova-plugin-nativestorage

Something like this?
export default function (key = 'vuex') {
  function getState (key) {
    return new Promise((resolve, reject) => {
      NativeStorage.getItem(key, resolve, reject)
    })
  }

  function setState (key, state) {
    return new Promise((resolve, reject) => {
      NativeStorage.setItem(key, state, resolve, reject)
    })
  }

  return function (store) {
    getState(key).then(state => {
      if (typeof state === 'object') {
        const merged = Object.assign({}, store.state, savedState)
        store.replaceState(merged)
      }
    })

    store.subscribe((mutation, state) => setState(key, state))
  }
}


Added missing keybinding for SetEquateAction

@caheckman this adds the 'E' keybinding to the SetEquateAction introduced in 1391e83ce9f63faf7fafc0d17b875d77729d5716. This way it will mirror the behavior of the action in the EquatePlugin.

Closed by 8ce66a52cdd1ae7a7cec4497ee8cd151d28b15a4


Doc Needs to Be Updated

Hi,
I'm going through this library and find out that the doc is out of date due to either not keeping up with the code evolution or something else... Would you mind to keep the API doc up-to-date? Also, would you like to add missing examples to those methods that don't have one? Thank you.

@ZheyangSong the API has been pretty stable for a while.
Feel free to list out the APIs that are missing examples and I can add examples.

Closing due to lack of activitiy


Update service-fabric-set-up-continuous-integration.md

Hi @CawaMS, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (cawa). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

#sign-off

@CawaMS - Please make updates in the private repo. You can pull these changes over to your local copy of the private repo using this command:
git pull https://github.com/CawaMS/azure-content.git patch-11


Add integration tests that run go.

Largely cribbed from @panicbit in  https://github.com/panicbit/rust-gob!

Note the boolean serialization test is failing. Not sure if it actually caught a bug or if it is a bug in my stuff yet:
---- ser::integration::test_bool stdout ----
        thread 'ser::integration::test_bool' panicked at '
Golang Expected:
{V:true}
Golang Decoded:
&{V:false}
', tests/ser/integration.rs:44:1

Actually, looks like it caught a legit bug 🐛 .
Manually dumping from Go gives me:
00000000  1e ff 81 03 01 01 0a 42  6f 6f 6c 53 74 72 75 63  |.......BoolStruc|
00000010  74 01 ff 82 00 01 01 01  01 56 01 02 00 00 00 05  |t........V......|
00000020  ff 82 01 01 00

Plugging it into a unit test:
#[derive(Serialize)]
struct BoolStruct {
    #[serde(rename = "V")]
    v: bool,
}

impl SchemaSerialize for BoolStruct {
    fn schema_register<S: Schema>(schema: &mut S) -> Result<S::TypeId, S::Error> {
        schema.register_type(
            Type::build()
                .struct_type("BoolStruct", 2)
                .field("V", TypeId::BOOL)
                .end(),
        )
    }
}

#[test]
fn bool_struct() {
    let mut buffer = Vec::new();
    {
        let mut stream = StreamSerializer::new(&mut buffer);
        stream.serialize(&BoolStruct { v: true }).unwrap();
    }
    assert_eq!(
        buffer,
        [
            0x1e, 0xff, 0x81, 0x03, 0x01, 0x01, 0x0a, 0x42, 0x6f, 0x6f, 0x6c, 0x53, 0x74, 0x72,
            0x75, 0x63, 0x74, 0x01, 0xff, 0x82, 0x00, 0x01, 0x01, 0x01, 0x01, 0x56, 0x01, 0x02,
            0x00, 0x00, 0x00, 0x05, 0xff, 0x82, 0x01, 0x01, 0x00,
        ].as_ref()
    );
}

Running gives me:
---- ser::bool_struct stdout ----
        thread 'ser::bool_struct' panicked at 'assertion failed: `(left == right)`
  left: `[30, 255, 129, 3, 1, 1, 10, 66, 111, 111, 108, 83, 116, 114, 117, 99, 116, 1, 255, 130, 0, 1, 1, 1, 1, 86, 1, 2, 0, 0, 0, 3, 255, 130, 0]`,
 right: `[30, 255, 129, 3, 1, 1, 10, 66, 111, 111, 108, 83, 116, 114, 117, 99, 116, 1, 255, 130, 0, 1, 1, 1, 1, 86, 1, 2, 0, 0, 0, 5, 255, 130, 1, 1, 0]`', tests/ser/mod.rs:484:5

Found the bug, fixed in https://github.com/srijs/rust-gob/pull/3. I cherry-picked that change to this branch and confirmed tests all pass.

Rebased. Not super happy with it but it all works. Will work to clean it up on an ongoing basis (like putting the go code in a template, etc)


[EUWE] Add missing icons for provider policies & compliance events

Backport of ManageIQ/manageiq-ui-classic#94 without miq_event_definition_decorator.rb that won't merge and is not yet necessary on euwe, because #11831 was euwe/no.
Verified this fixes missing icons in all the places that ManageIQ/manageiq-ui-classic#94 does (see there for screenshots).
Followup to ManageIQ/manageiq#11002,
BZ https://bugzilla.redhat.com/show_bug.cgi?id=1411364 (euwe)

@simaishi the master version landed, would be nice to merge before freezing euwe-2.
Just missing icons, as safe as it gets... (and yes I separately tested this on euwe)


Move settings outside of app

this allows us to make migration tools without needing to access app (doesn't work)
pr'd rather than merged as i know cmyui wants to review this first, just in case :)

there are other tools using manual DB_DSN still i believe

💃🏻 💃🏻 💃🏻 💃🏻 💃🏻 💃🏻 💃🏻


Sendgrid

Issue Summary
A summary of the issue and the environment in which it occurs. If suitable, include the steps required to reproduce the bug. Please feel free to include screenshots, screencasts, code examples.
Steps to Reproduce

This is the first step
This is the second step
Further steps, etc.

Any other information you want to share that is relevant to the issue being reported. Especially, why do you consider this to be a bug? What do you expect to happen instead?
Technical details:

sendgrid-nodejs Version: master (latest commit: [commit number])
Node.js Version: 4.1.2

Hi, it looks like maybe you omitted what problem you are experiencing in this issue?


same colors for different maze sizes

Is there any way to make the number of colors the same for all sizes of mazes? I find it very interesting to test generalization on more complex problems by learning on easier ones. However, to test the memory only (and not a representations), it is important that the colors be familiar to the agent (for example during transition from 9x9 to 15x15 maze).

So 9x9 has 3 objects/colors and 15x15 has 6 objects/colors. Do you suggest that 9x9 would have 3 objects but sample randomly from all 6 available colors?

@jurgisp Yeah, something in such direction, so that the problem remains the same (at least I believe that the task will not change globally and will not become more difficult), but there will be no representation shift after transfer to the new harder task. Maybe this can be an optional setting?

Yeah, I like the idea! It should be fairly easy to add such an option, I'll give it a go.

@Howuhh check out the PR and
python gui/run_gui.py --env "memory_maze:MemoryMaze-9x9-6CL-v0"

from the branch.

@jurgisp It's awesome, thanks! Seems to work nicely.


KYLIN-4101: set hive and spark job name when building cube

See: https://issues.apache.org/jira/browse/KYLIN-4101

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@0b38be9). Click here to learn what that means.
The diff coverage is 25%.

@@           Coverage Diff            @@
##             master    #764   +/-   ##
========================================
  Coverage          ?   25.7%
  Complexity        ?    6023
========================================
  Files             ?    1387
  Lines             ?   82834
  Branches          ?   11599
========================================
  Hits              ?   21292
  Misses            ?   59489
  Partials          ?    2053

Impacted Files
Coverage Δ
Complexity Δ

...pache/kylin/source/hive/GarbageCollectionStep.java
0% <0%> (ø)
0 <0> (?)

...org/apache/kylin/engine/spark/SparkExecutable.java
0% <0%> (ø)
0 <0> (?)

...lin/source/hive/RedistributeFlatHiveTableStep.java
0% <0%> (ø)
0 <0> (?)

...apache/kylin/source/hive/CreateMrHiveDictStep.java
0% <0%> (ø)
0 <0> (?)

...ylin/engine/spark/SparkBatchCubingJobBuilder2.java
0% <0%> (ø)
0 <0> (?)

...kylin/engine/spark/SparkBatchMergeJobBuilder2.java
0% <0%> (ø)
0 <0> (?)

...che/kylin/source/hive/CreateFlatHiveTableStep.java
0% <0%> (ø)
0 <0> (?)

...a/org/apache/kylin/rest/job/StorageCleanupJob.java
64.95% <100%> (ø)
31 <0> (?)

...a/org/apache/kylin/common/util/HiveCmdBuilder.java
79.48% <60%> (ø)
13 <1> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b38be9...99d1c44. Read the comment docs.

Pull Request Test Coverage Report for Build 4937

5 of 15 (33.33%)  changed or added relevant lines in 8 files are covered.
3 unchanged lines in 2 files lost coverage.
Overall coverage remained the same at 28.083%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

source-hive/src/main/java/org/apache/kylin/source/hive/CreateFlatHiveTableStep.java
0
1
0.0%

source-hive/src/main/java/org/apache/kylin/source/hive/CreateMrHiveDictStep.java
0
1
0.0%

source-hive/src/main/java/org/apache/kylin/source/hive/GarbageCollectionStep.java
0
1
0.0%

source-hive/src/main/java/org/apache/kylin/source/hive/RedistributeFlatHiveTableStep.java
0
1
0.0%

engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java
0
2
0.0%

engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchMergeJobBuilder2.java
0
2
0.0%

engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java
0
2
0.0%

Files with Coverage Reduction
New Missed Lines
%

engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkExecutable.java
1
0.0%

core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java
2
68.46%

Totals

Change from base Build 4935:
0.0%

Covered Lines:
23441

Relevant Lines:
83470

💛  - Coveralls


Provide download link for CSS

I don't want to build from source. Where can I find a compiled version, e.g. bootstrap-email.dist.css?

That's not how it works, it isn't just using a stylesheet, it embeds the styles in the html and rewrites the html into table layouts for email client compatibility. If you just want a css file to include in your emails you can use the css from regular Bootstrap.


Stored ImageStreams are invalid

When not using compression, libaff4 generates the wrong metadata for the relevant ImageStream. Eg:
aff4:compressionMethod aff4:NullCompressor ;
In this case it should not store a compressionMethod. From the spec v1
"Where compression is used within the Image Stream, the object must have a property aff4:compressionMethod set to a resource identifying the compression algorithm. Where there is no compressionMethod set, it is assumed that chunks are stored."

Reading that paragraph does not seem to suggest that this is incorrect behavior:

If we use compression we have to set the property
If the property is not set then we assume the null compressor (i.e. the default setting in null compressor).

We are setting the null compressor explicitly which is not forbidden. It is also clearer since we are less open to interpretations of the default value.

Ahh sorry - this commit actually fixes issue #94

So the main issue here is that aff4:NullCompressor is actually not defined in the spec at all. Instead the spec suggests that if the property is omitted it indicates that null compression is implied: i.e. if the writer intends to indicate no compression was used they should not include the property at all.
We have a couple of options:

Change the code to never emit a property if there is no compression at all. This is similar to the evimetry behavior. Evimetry currently chokes on the aff4:NullCompressor as an unknown compressor and the only way to imply no compression is to not have the property at all.

Extend the Evimetry tool to understand aff4:NullCompressor as an extension to the spec, while at the same time introducing the aff4:NullCompressor in the 1.1 spec revision. I feel it is better to have a way to properly specify no compression rather than relying on an absence of specification and an implied default value.

For c-aff4 we will continue recognizing the aff4:NullCompressor anyway so we can continue reading images which we produced in the past and specify it. While if the property is not specified at all we set aff4:NullCompressor as the default - this way we can continue reading Evimetry images too.

In a sense the issue is that Evimetry is more strict in the type of images it is willing to read so we need to produce images that follow the exact interpretation of the spec to make sure our images are properly supported by Evimetry readers. We should continue to be flexible in the type of images we can read though and accept either way if we can.
I feel that the path of least resistance is for us to change to increase interoperability - moving forward Evimentry can read our stored images which it can not now and we can read our stored images as normal either way (old ones or new ones). The discussion about the 1.1 standard is orthogonal and separate.

it turns out that we have been using the NullCompressor lexicon element for this in Evimetry all along - it is only missing from pyaff4. Given this is the case, I no longer think we need to take the spec so literally, and think it is safe to treat the NullCompressor as errata to the 1.0 spec and fix pyaff4.


Music keep playing when i boot cores and even during gameplay.

Just tried it and the mp3 music continue to play when i go to a core.

That's weird, I'm not sure why it wouldn't work on another mister.
Just so you don't go crazy, you can either delete or the stuff from the music folder or comment out the startup line in /media/fat/linux/user-startup.sh to shut it up
Does this still consistently happen for you after you've rebooted as well? And do any of the bgm_play/stop/skip script work for you?
Thanks! And sorry it's not working properly for you

The way it detects if a core has been launched is a file called /tmp/CORENAME on the mister which get written to each time a core is launched, so it sounds like the thing that watches changes on that isn't working properly. Maybe can can double check if that file exists for you
I will write in some logging stuff later for the script so I can troubleshoot better

Glad to hear it's working for you now! Thank you
I'm going to keep this open for now if that's ok. I will see if I can reproduce the issue

I have 5 mp3 files.

Music keeps playing when a core is loaded.
play/stop/skip scripts do not work.
socat[1229] E connect(5, AF=1 "/tmp/bgm.sock", 15): No such file or directory

And yet:
/tmp# ls
bgm.sock  dbus	    ML_FSYNTH	 script  subsys      wtmp
CORENAME  messages  resolv.conf  sshd	 UART_SPEED
/tmp# ls -ltr
total 64
-rwxr-xr-x 1 root root   165 Apr 13 14:31 script
srwxr-xr-x 1 root root     0 Apr 13 14:32 bgm.sock
-rw-r--r-- 1 root root  1920 Apr 13 14:32 wtmp
-rw-r--r-- 1 root root 44424 Apr 13 14:32 messages
-rw-r--r-- 1 root root     1 Apr 13 14:33 UART_SPEED
-rw-r--r-- 1 root root     0 Apr 13 14:33 ML_FSYNTH
-rw-r--r-- 1 root root     4 Apr 13 14:33 CORENAME
drwxr-xr-x 2 root root    60 Apr 13 14:33 dbus
drwxr-xr-x 2 root root    60 Apr 13 14:33 subsys
-rw-r--r-- 1 root root   153 Apr 13 14:33 resolv.conf
-rw------- 1 root root     0 Apr 13 14:33 sshd

Thanks for the report. I'm still trying to investigate this one, I've seen the socket file issue reported in a few different places now but it's very difficult for me to troubleshoot with just 1 mister that doesn't have the problem
I'm wondering if it's something to do with the /tmp folder because almost all issues so far have been to do with the files there. Could you paste the contents of your /etc/fstab here too please?

Unfortunately, everything is working 100% (auto-start, play/stop/skip scripts, stops when core is launched) ATM.
Will post additional info when it is not working (using the latest as of April 14).

Music keeps playing after launching a core...
/etc# cat fstab
# <file system>	<mount pt>	<type>	<options>	<dump>	<pass>
/dev/root	/		ext4	rw,noauto,noatime,nodiratime	0	1
proc		/proc		proc	defaults	0	0
devpts		/dev/pts	devpts	defaults,gid=5,mode=620,ptmxmode=0666	0
tmpfs		/dev/shm	tmpfs	mode=0777	0	0
tmpfs		/tmp		tmpfs	mode=1777	0	0
tmpfs		/run		tmpfs	mode=0755,nosuid,nodev	0	0
sysfs		/sys		sysfs	defaults	0	0
tmpfs		/var/lib/samba	tmpfs	mode=1777	0	0

/tmp# ls -ltr
total 60
drwxr-xr-x 2 root root    60 Apr 14 11:44 dbus
drwxr-xr-x 2 root root    60 Apr 14 11:44 subsys
-rw-r--r-- 1 root root   153 Apr 14 11:44 resolv.conf
-rw------- 1 root root     0 Apr 14 11:44 sshd
-rw-r--r-- 1 root root     1 Apr 14 11:44 UART_SPEED
-rw-r--r-- 1 root root     0 Apr 14 11:44 ML_FSYNTH
-rw-r--r-- 1 root root     6 Apr 14 11:44 CORENAME
srwxr-xr-x 1 root root     0 Apr 14 11:45 bgm.sock
-rw-r--r-- 1 root root  1152 Apr 14 11:45 wtmp
-rw-r--r-- 1 root root 45054 Apr 14 11:45 messages
/tmp# cat CORENAME
NEOGEO

@CharlieYankeeUniformLima do you have any startup sounds running? I've seen some issues when people have a startup sounds run long enough to end after a core is launched and that messes up monitoring for core launches. I'll be putting in a fix for that soon
Also, when you say the the music keeps playing after launch, does that mean it never stops? Or would it stop and start like normal after your first core launch
Thanks again for troubleshooting

I have a startup sound. I had a short sound file and it did not cause any problems. The sound stopped playing when I launched a core.
With a longer startup sound (5min), it kept playing after launching a core and kept looping until I switched core.
This is with the first core loaded and the music is playing/looping:
/tmp# ls -ltr
total 60
drwxr-xr-x 2 root root    60 Apr 19 09:20 dbus
drwxr-xr-x 2 root root    60 Apr 19 09:20 subsys
-rw-r--r-- 1 root root   153 Apr 19 09:20 resolv.conf
-rw------- 1 root root     0 Apr 19 09:20 sshd
-rw-r--r-- 1 root root     0 Apr 19 09:20 ML_FSYNTH
-rw-r--r-- 1 root root     6 Apr 19 09:20 CORENAME
-rw-r--r-- 1 root root     1 Apr 19 09:20 UART_SPEED
-rw-r--r-- 1 root root  1152 Apr 19 09:20 wtmp
-rw-r--r-- 1 root root 44409 Apr 19 09:20 messages
/tmp# cat CORENAME
NEOGEO

/tmp# cat /etc/fstab
# <file system>	<mount pt>	<type>	<options>	<dump>	<pass>
/dev/root	/		ext4	rw,noauto,noatime,nodiratime	0	1
proc		/proc		proc	defaults	0	0
devpts		/dev/pts	devpts	defaults,gid=5,mode=620,ptmxmode=0666	0
tmpfs		/dev/shm	tmpfs	mode=0777	0	0
tmpfs		/tmp		tmpfs	mode=1777	0	0
tmpfs		/run		tmpfs	mode=0755,nosuid,nodev	0	0
sysfs		/sys		sysfs	defaults	0	0
tmpfs		/var/lib/samba	tmpfs	mode=1777	0	0

After switching core:
/tmp# ls -ltr
total 60
drwxr-xr-x 2 root root    60 Apr 19 09:20 dbus
drwxr-xr-x 2 root root    60 Apr 19 09:20 subsys
-rw-r--r-- 1 root root   153 Apr 19 09:20 resolv.conf
-rw------- 1 root root     0 Apr 19 09:20 sshd
-rw-r--r-- 1 root root  1152 Apr 19 09:20 wtmp
srwxr-xr-x 1 root root     0 Apr 19 09:24 bgm.sock
-rw-r--r-- 1 root root     1 Apr 19 09:27 UART_SPEED
-rw-r--r-- 1 root root     0 Apr 19 09:27 ML_FSYNTH
-rw-r--r-- 1 root root     4 Apr 19 09:27 CORENAME
-rw-r--r-- 1 root root 44753 Apr 19 09:27 messages
/tmp# cat CORENAME
SNES
/tmp# cat /etc/fstab
# <file system>	<mount pt>	<type>	<options>	<dump>	<pass>
/dev/root	/		ext4	rw,noauto,noatime,nodiratime	0	1
proc		/proc		proc	defaults	0	0
devpts		/dev/pts	devpts	defaults,gid=5,mode=620,ptmxmode=0666	0
tmpfs		/dev/shm	tmpfs	mode=0777	0	0
tmpfs		/tmp		tmpfs	mode=1777	0	0
tmpfs		/run		tmpfs	mode=0755,nosuid,nodev	0	0
sysfs		/sys		sysfs	defaults	0	0
tmpfs		/var/lib/samba	tmpfs	mode=1777	0	0

I'm getting this too. Short startup sound, and then a random song from 5 files. Plays when a core is loaded and keeps playing when a game is loaded.

I'm getting this too. Short startup sound, and then a random song from 5 files. Plays when a core is loaded and keeps playing when a game is loaded.

Hi. Sorry for the delay. Is this still happening for you? If so, can you please change the debug setting in your bgm.ini file to "debug = yes", restart your mister, launch a game (to reproduce the problem happening) and then paste the contents of your /tmp/bgm.log file here. You'll need to grab the file while your mister is still on, it gets deleted on reboot.

Well maybe I misunderstand what is supposed to happen. I assumed that allowing the music to play when a core was loaded meant just when you're in the core menus and once a ROM is loaded that the music would stop not play over top of it.

That's not how it works sorry. It just means it will continue playing all the time, including when games are played. I would like to set it up how you've described, but unfortunately there's no way for me to tell if a game is actually running from the linux side
You might like my games menu script though. It will let you launch roms from the main mister menu, so you can listen to music while you browse

That's too bad. I should note though that I originally have the core music play back disabled, but it still occasionally would keep playing. I've since disabled core playback again, so if it happens again, I'll try and reproduce and enable the logging.


Could I get the error info from CMakeApp (RDT-624)

Is there any way to get error messages when I compile app by using CMakeApp.

Hi @feizi, thank you for the question.
when you're building it by (a simple example, assume running it under $IDF_PATH)
CMakeApp('examples/get-started/hello_world', 'esp32').build()

then the cmake output should be printed to the stdout by default.
Are you requesting something else?

I wonder if I can get the parsed error messages.
For example:
{messgae: "expected ';' before '}' token", position:"main.c:7:29:"
...

Okay, I see. Thank you. I'll create a feature request and move the current issue to discussion.