Enable partial masking of IP addresses in access logs

This PR is inspired by mod_log_ipmask for apache. It was written from scratch by myself (as should be obvious from the git history), i. e. is not based on the actual code of mod_log_ipmask.
It adds an optional format parameter to the %a / %h placeholders in the accesslog.format config option. The parameter can specify how many trailing bits of the client IP address to mask (i. e. set to 0). Separate values can be set for IPv4 and IPv6. Both default to 0, meaning no masking, i. e. the behaviour is compatible with previous versions.

It was written from scratch by myself

based on what?  (I'm not challenging; just asking.)  Please add to the above to one of the commit messages and point to the text specification/description of what this does.  I can see that the "specification" is not complicated and is an extension to already existing accesslog format specifiers, and apologize for the formalities, but precautions are necessary.
Please use https://wiki.lighttpd.net/mod_accesslog instead of https://redmine.lighttpd.net/projects/lighttpd/wiki/Mod_accesslog
<netinet/in.h> unfortunately is not as portable as we'd all like.  Please replace that and <sys/socket.h> with the include of the local abstraction #include "sys-socket.h"
mask() is a poor name for a function, even a local one.  Are you masking IP bits?  Are you masking upper IP bits or lower IP bits?  Can the function name better describe what it is doing (in brief) in two or three words separate by underscore?  (Yes, I can read what the code does, despite these questions. )
Is there a convention for masking the lower bits instead of masking the upper bits?  The lower bits might be better for matching up the same (slightly anonymized) client.  Is this masking permitted by GDPR?  For what reasons are you masking?  Can you provide pointers to the specifications/guidance?  Is masking reasonable?  Should the IP instead be passed through a hash function such as SHA256?
I'll try to review more later this weekend.  I hope there is a way to do this with less code and hopefully without allocation.
sock_addr_is_addr_eq_bits() in sock_addr.c might be referenced, and we might consider a function in sock_addr.c rather than cut-n-pasting code into mod_accesslog.c, which ideally should not care about IPv4, IPv6, unix domain socket distinctions, as should be apparent by you having to add less-portable headers into mod_accesslog.c.

Yes, I know I named a function pointer esc() for the local variable, so asking you to rename mask() might seem unfair.
The code in the for() loop in log_access_record() is intentionally tight, so you additions into log_access_record() should be pulled out into a better-named mask(), (accesslog_append_remote_addr()?) which takes various parameters including b, the buffer into which to write the address, rather than the tmp that you created.  I'm nearly certain that we'll end up marking that function __attribute_noinline__

It was written from scratch by myself

based on what?

Not sure I understand the question. I have sufficient programming background to write code given a task like "mask lower bits of an ip address".
To be clear, I knew the mod_log_ipmask code before implementing this for lighttpd. If that means I cannot contribute here, then so be it. No worries.

Please add to the above to one of the commit messages and point to the text specification/description of what this does.

Done. I included the "specification" into the message. I couldn't find a place in the repo where to add it. I guess it will be added to the wiki at some point.

Please use https://wiki.lighttpd.net/mod_accesslog instead of https://redmine.lighttpd.net/projects/lighttpd/wiki/Mod_accesslog

Done.

<netinet/in.h> unfortunately is not as portable as we'd all like. Please replace that and <sys/socket.h> with the include of the local abstraction #include "sys-socket.h"

Done (but irrelevant after refactoring).

mask() is a poor name for a function, even a local one. Are you masking IP bits? Are you masking upper IP bits or lower IP bits? Can the function name better describe what it is doing (in brief) in two or three words separate by underscore?

Done.

Is there a convention for masking the lower bits instead of masking the upper bits? The lower bits might be better for matching up the same (slightly anonymized) client. Is this masking permitted by GDPR? For what reasons are you masking? Can you provide pointers to the specifications/guidance? Is masking reasonable? Should the IP instead be passed through a hash function such as SHA256?

Here's a pointer to something official: https://www.datenschutzzentrum.de/artikel/575-IP-Adressen-und-andere-Nutzungsdaten-Haeufig-gestellte-Fragen.html
Obviously, this applies to Germany. IANAL, but I think the principle applies to the GDPR in general.
An IP address is considered personally identifiable information because it is possible to find out who was using it at a given time with the help of the ISP who owns the address. By stripping off a sufficient number of bits from the address this possibility no longer exists, which means the remaining bits can be considered anonymous.
Stripping off the rightmost bits instead of left has two main reasons:

since IP address blocks are handed out to ISPs per prefix, the prefix still contains some information about the (geographic) origin of a request, which is probably one of the main reasons for logging IP addresses at all.
For IPv6 addresses, the lower bits are in some cases derived from hardware IDs (MAC address). This alone is sufficient to identify a specific device (and thus its owner). Therefore, stripping off higher bits would not be sufficient to anonymize the address.

Hashing is not sufficient, because at least for v4 addresses it is feasible to find an IP address for a given hash by brute-forcing the IP address range. Also, hashing has the downside of losing the address prefix and thus the geographic origin of a request.

sock_addr_is_addr_eq_bits() in sock_addr.c might be referenced, and we might consider a function in sock_addr.c rather than cut-n-pasting code into mod_accesslog.c, which ideally should not care about IPv4, IPv6, unix domain socket distinctions, as should be apparent by you having to add less-portable headers into mod_accesslog.c.

Done.

The code in the for() loop in log_access_record() is intentionally tight, so you additions into log_access_record() should be pulled out into a better-named mask(),

Done.

It was written from scratch by myself

based on what?

Not sure I understand the question. I have sufficient programming background to write code given a task like "mask lower bits of an ip address".

Haha.  I do not think you woke up one day and decided "You know what I need?  I need to mask IP bits in my logs!"  :)
You answered the question with your response to my more detailed questions about regulations and how masking the IP address is permitted by (at least some) official guidelines.
Regarding SHA256 hash, yes, I should have suggested that the hash be salted and the salt changed periodically, similar to what lighttpd does multiple times per day with the TLS session ticket encryption key.  Then again, you raise a good point about losing the utility of retaining the high bits of the address for offline analytics.

Some technical questions for discussion (before asking you to make any further code changes):

Any specific existing convention on which you based the format specifier syntax extension (%{v[46]:numbits[,...]}a)?  Is there a necessity for this to be configurable?  Are there regulations saying IP mask should be a minimum of XX bits?  The article you referenced suggests that masking the lower octet of IPv4 addresses is sufficient.  (I don't agree, but that's not the point.)  If there is a convention or standard, should we more simply provide an option to do "suggested" masking?
mod_accesslog is not the only place in lighttpd which deals with IP addresses and logging.  Among others, there is also the lighttpd error log(s), which is why I mentioned in https://redmine.lighttpd.net/boards/3/topics/10949 that I am generally of the opinion that arbitrary log munging should be done by a piped logger.

Regarding use of lighttpd sock_addr_cache_inet_ntop_copy_buffer().  That is currently a small 4-element cache currently used only in one place in the lighttpd base codebase (connections.c).  Leveraging the same cache for masked IP addresses reuses the code, which is good, but halves the size of effective cache, and adds some unnecessary work since the masked entries will (almost) never match for connections.c, and the unmasked entries will never match for mod_accesslog use.  The cache as is currently written is not thread-safe, but lighttpd is not threaded.  I could minimally extend sock_addr_cache_inet_ntop_copy_buffer() to take a cache object on which to operate, and have connections.c and mod_accesslog provide their own cache objects.  Alternatively, the use of the cache could be removed from mod_accesslog in this PR, similar to how the cache is not used in http_cgi.c when setting SERVER_ADDR.  (On the other hand, if sock_addr_cache_inet_ntop_copy_buffer() is extended, then http_cgi.c might also be extended to have its own cache object.)
A quick internet search suggests to me that IP masking in-server (e.g. not performed by piped loggers or similar) is frequently implemented via string matching / regex on the stringified IP address.  Note: I have not measured the performance of potentially (limited, non-configurable) masking based on a string match of the already stringified IP address, versus masking the binary address and then stringifying using inet_pton(), potentially mediated by a small cache.  String flinging of the stringified IP could be implemented in lighttpd mod_magnet and while reasonably fast and possible to do today without patching lighttpd, I do believe that the PR here will be faster.

This idiom would be more likely to minimize the overhead for existing lighttpd use, without the extended format specification.  accesslog_append_addr_masked() would be marked __attribute_noinline__ and would have to handle non-IP strings (e.g. AF_UNIX), but there are existing routines in sock_addr.h to help.
-    buffer_append_string_buffer(b, r->dst_addr_buf);
+    if (!f->opt)
+        buffer_append_string_buffer(b, r->dst_addr_buf);
+    else
+        accesslog_append_addr_masked(b, r->dst_addr, f->opt);
     break;

While I am a fan of C99 designated initializers, lighttpd is still built by some people for very old machines with ancient compilers, or on systems with braindead compilers (historically MS C++ compilers).
Your sock_addr_mask_lower_bits() is nicely reusable.
Depending on other discussions above, we might combine this and sock_addr_cache_inet_ntop_copy_buffer() into a specialized sock_addr_cache_inet_ntop_masked_append_buffer()

Any specific existing convention on which you based the format specifier syntax extension (%{v[46]:numbits[,...]}a)?

None that I know. It felt natural.

Is there a necessity for this to be configurable? Are there regulations saying IP mask should be a minimum of XX bits? The article you referenced suggests that masking the lower octet of IPv4 addresses is sufficient. (I don't agree, but that's not the point.) If there is a convention or standard, should we more simply provide an option to do "suggested" masking?

There is no law specifying precisely how many bits must be removed in order to sufficiently anonymize an address, and even if there was the value might vary between jurisdictions. I therefore think it should be configurable.

mod_accesslog is not the only place in lighttpd which deals with IP addresses and logging. Among others, there is also the lighttpd error log(s), which is why I mentioned in https://redmine.lighttpd.net/boards/3/topics/10949 that I am generally of the opinion that arbitrary log munging should be done by a piped logger.

Right. mod_log_ipmask ignores the error log as well IIRC. In my understanding, an IP address in the error log is more likely to be a case of legitimate interest for the server operator. The article I linked above specifically mentions that it is acceptable to keep unmasked IP addresses for security purposes for up to 7 days.
Again, this might vary between jurisdictions, so maybe we should look for a way to apply the masking in a more general way.

Regarding use of lighttpd sock_addr_cache_inet_ntop_copy_buffer(). That is currently a small 4-element cache currently used only in one place in the lighttpd base codebase (connections.c).

Yes, I noticed. My first impression was that the cache is most likely a case where Knuth's premature optimization rule should have been applied. :-)
IMO in a high-load installation the cache will be of little use because it is too small, so entries will be overwritten before they could be re-used. In a low-load installation the cache might be hit more often, but the few extra cycles for stringifying the address each time wouldn't hurt.

A quick internet search suggests to me that IP masking in-server (e.g. not performed by piped loggers or similar) is frequently implemented via string matching / regex on the stringified IP address.

That's easy to do when you just want to drop the last octet from an IPv4 address, but more difficult if you want to strip a different number of bits. It's also more difficult for IPv6 addresses.

Again, this might vary between jurisdictions, so maybe we should look for a way to apply the masking in a more general way.

Sorry, that falls under the category of "no".  Such policy enforcement, which might need to look for other PII (personally identifiable information), too, should be done by a filter before persistent storage of that information, e.g. by a piped logger.  The recommendation for a separate policy enforcement layer prior to persistent storage remains the same as when I posted in 2018 in https://redmine.lighttpd.net/boards/2/topics/8097 around the time that GDPR came into force.
Since mod_accesslog is optional and optionally used for analytics, this PR is being considered for convenience of those using mod_accesslog and trying to comply with GDPR.  I do not wish to expand the scope of this PR.

Any specific existing convention on which you based the format specifier syntax extension (%{v[46]:numbits[,...]}a)?

None that I know. It felt natural.

What should be the default if only one of IPv4 or IPv6 mask is specified?
If the user-interface is confusing to us, then it will very likely be confusing to many people who try to use it for GDPR compliance.
I am still mulling over this interface, as it is a user-visible change, and if we do it right, then others might use it as a reference and convention.  However, I have to refresh myself on IPv6 notation and all that junk to have a more informed opinion.  My questions above are me dipping my toe in.  I had the preliminary thought of %{~24}a or some other syntax that looks to be the inverse of CIDR notation as the number of bits to mask, and I am considering the limitation of masking up to 32-bits and whether or not that would be sufficient for IPv6 -- and whether or not the IP masking interface must allow specifying separate IPv4 and IPv6 masks.

This approach would be more likely to minimize the overhead for existing lighttpd use

Done.

lighttpd is still built by some people for very old machines with ancient compilers

Done.

please remove the use of the cache in your PR

Done. Thanks for the explanation.

What should be the default if only one of IPv4 or IPv6 mask is specified?
I am considering the limitation of masking up to 32-bits and whether or not that would be sufficient for IPv6
and whether or not the IP masking interface must allow specifying separate IPv4 and IPv6 masks.

Good questions. And they're related.
IMO 32 bits is not sufficient for IPv6 since I've seen ISPs handing out /64s per customer. An alternative could be to take a value of n for IPv4 and n+64 for IPv6 (at least for n > 0).

Note: my time is more limited the next few days, so follow-ups may be delayed.
Continuing discussions: masking interface (mod_accesslog format specifier) and masking requirements
Regarding masking requirements, while lighttpd mod_accesslog could be configurable, is there a strong reason not to do the same as is done by Google Analytics?  https://support.google.com/analytics/answer/2763052
(remove last octet for IPv4, remove last 10 octets for IPv6)
This is repeated on more than a few other sites.  Some examples:
https://www.mediawiki.org/wiki/GDPR_(General_Data_Protection_Regulation)_and_MediaWiki_software

Mask IP addresses of unregistered users, by removing the last byte of IPv4 address (maybe by setting it to 0, for example, <IP_ADDRESS> would become <IP_ADDRESS>), or the last 80 bits in case of a IPv6 address. This is what Google Analytics does for being compliant: IP anonymization on analytics.

https://www.cookieyes.com/blog/ip-anonymization-in-google-analytics-for-gdpr-compliance/

IP anonymization, also known as IP masking, is a method of replacing the original IP address with one that cannot be associated with or traced back to an individual user. This can be done by setting the last octet of IPV4 addresses or the last 80 bits of IPv6 addresses to zeros.

https://github.com/ankane/ip_anonymizer

This is the approach Google Analytics uses for IP anonymization:
For IPv4, set the last octet to 0
For IPv6, set the last 80 bits to zeros

Since lighttpd r->dst_addr_buf is a normalized IP string, string flinging to drop the bottom octet is trivial, and removing the bottom 8 octets for IPv6 is doable.  Is this something we should consider?  If we did, then the interface to %{...}a would be different, e.g. %{anon}a or %{gdpr}a (but that may be too bold a statement)  If this were done, could we ignore distinctions for IPv6 6in4, 6to4, 6over4?
If we develop a reasonable degree of confidence that doing X masking satisfies GDPR, then I would prefer a simpler interface.

Lightly tested string flinging masking of IPv4 final octet (8 bits) and IPv6 final 10 octets (80 bits):
__attribute_noinline__
static void
accesslog_append_remote_addr_masked (buffer *b, request_st *r)
{
    /* r->dst_addr_buf is normalized and valid string; operate on known valid */
    const char * const s = r->dst_addr_buf->ptr;
    uint32_t i = 0;
    switch (sock_addr_get_family(r->dst_addr)) {
      case AF_INET:
        /* IPv4: mask final octet (8 bits) of address */
       #ifdef __COVERITY__
        force_assert(buffer_clen(r->dst_addr_buf) > 2);
        force_assert(strchr(s, '.') != NULL);
       #endif
        for (i = buffer_clen(r->dst_addr_buf)-1; s[--i] != '.'; ) ;
        buffer_append_str2(b, s, i+1, CONST_STR_LEN("0"));
        break;
     #ifdef HAVE_IPV6
      case AF_INET6:
        /* IPv6: mask final 10 octets (80 bits) of address */
        /* Note: treat string starting with "::..." as "::" even if "::x:..."
         * (rather than special-casing since I do not care about "::x:...") */
        for (int j = 0; s[i] != ':' || ((j += 2) != 6 && s[i+1] != ':'); ++i) ;
        buffer_append_str2(b, s, i+1, CONST_STR_LEN(":"));
        break;
     #endif
      default:
        buffer_append_buffer(b, r->dst_addr_buf);
        break;
    }
}

Regarding masking requirements, while lighttpd mod_accesslog could be configurable, is there a strong reason not to do the same as is done by Google Analytics? https://support.google.com/analytics/answer/2763052
This is repeated on more than a few other sites.

I wasn't aware of that. It's been quite a few years since I last researched the subject.
I found this recommendation to strip at least 88 bits in IPv6 addresses, but it predates GDPR.
I don't see a technical reason not to follow Google's approach. However, choosing fixed numbers means making a decision with potential legal consequences on behalf of your users. Not sure if you want to do that, although I believe the license terms would protect us from these consequences.

Since lighttpd r->dst_addr_buf is a normalized IP string, string flinging to drop the bottom octet is trivial, and removing the bottom 10 octets for IPv6 is doable. Is this something we should consider? If we did, then the interface to %{...}a would be different, e.g. %{anon}a or %{gdpr}a (but that may be too bold a statement).

Seems reasonable (but I agree that "%{gdpr}a" may not be a good idea).

If this were done, could we ignore distinctions for IPv6 6in4, 6to4, 6over4?

6in4 and 6over4 are about routing not distinguishable on the endpoints, these can be ignored.
6to4 is deprecated and can probably be treated like regular IPv6, for simplicity. It would be covered automatically if we followed the above recommendation to strip 88 bits from IPv6 addresses (leaving 16 bit 6to4 prefix and 24 bits of the IPv4 address).
Mapped IPv4 we cannot ignore, because that would mean IPv4 addresses are always logged as '::' when server.v4mapped = "enable" is configured.

If we develop a reasonable degree of confidence that doing X masking satisfies GDPR, then I would prefer a simpler interface.

I agree.

However, choosing fixed numbers means making a decision with potential legal consequences on behalf of your users. Not sure if you want to do that, although I believe the license terms would protect us from these consequences.

Are you a lawyer?  If you are not a lawyer, please do not present any opinions which might be mistaken as legal opinions.  I have discarded your entire statement without consideration.
The question/answer for IP masking in Google Analytics has been active for at least a decade (10 years) as of next month.
https://web.archive.org/web/20130405085532/https://support.google.com/analytics/answer/2763052
As I mentioned further above, in order for users to reliably do the right thing, the interfaces should be simple and helpful.
Providing a fully-configurable interface to mask IPv4 and IPv6, and requiring the user to separately fill in the values for IPv4 and IPv6 is flexible, but definitely not as simple as it could be.  Absent explicit, official government documents indicating otherwise, matching the IP masking behavior widely used on the internet for years is simple and likely the best thing to do for those trying to comply with the government regulations.
Therefore, I think %{anon}a or %{mask}a is the direction we should go in lighttpd mod_accesslog.

Regarding masking requirements, while lighttpd mod_accesslog could be configurable, is there a strong reason not to do the same as is done by Google Analytics? https://support.google.com/analytics/answer/2763052

Same question as before.  If you think lighttpd should mask 88 bits from IPv6 instead of masking 80 bits, then please provide more justification why that is better than doing the same as Google Analytics has done for the past decade.  The link you provided which suggested masking 88 bits is not convincing since as you note, the document predates the GDPR and the link you provided is to the wayback machine, not to a live document.

I pushed an experimental commit to my dev branch using %{anon}a and extending my code posted above to handle v4mapped.
See top commit at https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/personal/gstrauss/master
currently: https://git.lighttpd.net/lighttpd/lighttpd1.4/commit/e3374b2c6821119d82dc33c3f5338e7b48354ac9

Thanks. I'm closing this PR in favour of your change.
Your code lgtm. I found the line
for (int j=0; s[i] != ':' || ((j+=2) != 6 && s[i+1] != ':'); ++i) ;

a bit hard to understand, maybe this version would be more readable:
for (int colons=0; s[i] != ':' || ((++colons) != 3 && s[i+1] != ':'); ++i) ;

Also, for consistency you might want to add a force_assert in the V6 branch too to ensure there is either a :: or at least 3 colons in the address.

for (int colons=0; s[i] != ':' || ((++colons) != 3 && s[i+1] != ':'); ++i) ;

Yes, slightly clearer, but does not fit in 80 chars, so I'll keep the single line as-is.  This would fit in two lines:
int colons = 0;
while (s[i] != ':' || ((++colons) != 3 && s[i+1] != ':')) ++i ;

Were the code in question many lines, then a better named variable is much more desirable.  The comments directly above the one line say this masks 80 bits.  I could expand the comment to indicate how the existing one line is doing it, describing how 6 octets are 48-bits of a 128-bit address, masking 80 bits, but at some point comments should not describe that i += 1 says to add 1 to i.  Comments should generally describe why or what, not how, unless the algo is complex.  In this case, I prefer counting octets (or even nibbles) on the stringified IP.

Also, for consistency you might want to add a force_assert in the V6 branch too to ensure there is either a :: or at least 3 colons in the address.

The top of the func is commented: /* r->dst_addr_buf is normalized and valid string; operate on known valid */
The asserts are unnecessary given that the input has already been normalized.  The force_assert() in the IPv4 case is pre-emptively providing the hint to coverity, and is not active during normal builds.

Thank you for the discussion.


The led of my TL-WN722N-V2 doesn't turn on

uname -a

Linux moe 4.10.0-33-generic #37~16.04.1-Ubuntu SMP Fri Aug 11 14:07:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a

No LSB modules are available.
Distributor ID:	LinuxMint
Description:	Linux Mint 18.2 Sonya
Release:	18.2
Codename:	sonya

Led is now switched ON permanently to switch it off follow Readme. Please make and install repository again !


Missiles + entity driving

Missiles which are parented to the rack can be moved around with the entity drive feature.  Need to make sure they can't be moved at all.

Missiles on racks seem to be non solid on the server (physgun beam goes through) but not on the client (smartsnap grid shows). Perhaps add a clientside hack to make it non-solid there aswell, or use the CanDrive hook: http://wiki.garrysmod.com/page/SANDBOX/CanDrive

Check out 00cf9c36 + b7d7d7c6


SetupFinal: Do not display seedwords with comma

Response from blitz api:

This is also shown in the webui:

Should be cleaned from commas

If you can remove the comma on the WebUI side that would be best.


Made the User preferences data entity implement UserProfile interface

created user profile interface to better follow CA
made t the User preferences data entity implement UserProfile interface
made the profile DAO use user profile interface instead to follow CA
modified any related references
Also added some more test of the signup view

@kanishsharma You should probably take a look at this


Impossible to update homebridge-harmony@latest

Hi guys
I'm trying to update the package using commande
sudo npm install homebridge-harmony@latest
However I got an error and I don't understand how to fix it (see bellow)
I also attached the log file for the error
Thanks in advance for your support, I notice you fix the latest issue I've open 👍 🤘 you rock !
2019-02-01T18_42_46_073Z-debug.log

serveur:~ loris$ sudo npm install homebridge-harmony@latest
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/chnl-e6ba2fdd/es5'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/har-schema-1411984e/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/has-c09b9c62/src'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/performance-now-2431b5a3/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/promise-controller-4fe76d90/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/har-validator-605891bd/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/flat-options-cb8429bb/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/delayed-stream-1e0b70ef/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/tough-cookie-351286e2/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/has-c09b9c62/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/json-schema-traverse-7899032d/spec'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/typedarray-to-buffer-b985416f/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/combined-stream-20d0bafc/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/fast-json-stable-stringify-0271a217/benchmark'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/json-stringify-safe-e93c4c4b/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/promise-controller-4fe76d90/scripts'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/asynckit-7100a7bb/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/ajv-c1cd4f60/dist'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/yaeti-291b3705/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/has-symbols-1645e662/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/json-schema-traverse-7899032d/spec/fixtures'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/performance-now-2431b5a3/src'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/function-bind-cef2eba5/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/define-properties-156eb3fe/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/promise-controller-4fe76d90/src'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/qs-b60ae602/dist'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/object-keys-d9663346/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/chnl-e6ba2fdd/src'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/fast-json-stable-stringify-0271a217/example'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/has-symbols-1645e662/test/shams'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/is-symbol-ecad6a70/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/form-data-a61299ff/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/websocket-429502a3/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/performance-now-2431b5a3/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/qs-b60ae602/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/promise-controller-4fe76d90/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/promise.prototype.finally-8ea89141/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/uri-js-d79124fd/dist/es5'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/uuid-50435e86/bin'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/es-to-primitive-c8d94d2b/helpers'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/performance-now-2431b5a3/test/scripts'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/promise-controller-4fe76d90/test/specs'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/request-93d731e3/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/fast-json-stable-stringify-0271a217/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/psl-9bb3f3cc/data'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/uuid-50435e86/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/websocket-as-promised-5413c381/dist'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/es-to-primitive-c8d94d2b/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/chnl-e6ba2fdd/types'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/es-abstract-7dfdd777/helpers'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/psl-9bb3f3cc/dist'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/qs-b60ae602/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/ajv-c1cd4f60/lib'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/es-abstract-7dfdd777/operations'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/websocket-as-promised-5413c381/scripts'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/websocket-as-promised-5413c381/src'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/ajv-c1cd4f60/lib/compile'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/websocket-429502a3/src'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/uri-js-d79124fd/dist/esnext'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/websocket-as-promised-5413c381/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/websocket-429502a3/vendor'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/es-abstract-7dfdd777/test'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/websocket-as-promised-5413c381/test/specs'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/nan-24786827/doc'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/ajv-c1cd4f60/lib/dot'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/uri-js-d79124fd/dist/esnext/schemes'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/es-abstract-7dfdd777/test/helpers'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/websocket-as-promised-5413c381/types'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/nan-24786827/tools'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/uri-js-d79124fd/src'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/uri-js-d79124fd/src/schemes'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/ajv-c1cd4f60/lib/dotjs'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/uri-js-d79124fd/tests'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/ajv-c1cd4f60/lib/refs'
npm WARN tar EISDIR: illegal operation on a directory, open '/Users/loris/node_modules/.staging/ajv-c1cd4f60/scripts'
npm WARN enoent ENOENT: no such file or directory, open '/Users/loris/package.json'
npm WARN loris No description
npm WARN loris No repository field.
npm WARN loris No README data
npm WARN loris No license field.
npm ERR! path /Users/loris/node_modules/request/node_modules/sshpk/bin/sshpk-conv
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '/Users/loris/node_modules/request/node_modules/sshpk/bin/sshpk-conv'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent
npm ERR! A complete log of this run can be found in:

don't really know what happens there. I have jsut published through npm 0.2.0 with beta tv platform support

update to latest 0.2.0 fix the issue
Closed


chore: Add size labels to GitHub configuration

Pull Request
Description
This change adds new labels to categorise pull requests based on their size. The following size labels have been introduced:

size/XS: Extra Small Pull Request (colour: 00ff22)
size/S: Small Pull Request (colour: 03a319)
size/M: Medium Pull Request (colour: f2bb05)
size/L: Large Pull Request (colour: fc7e08)
size/XL: Extra Large Pull Request (colour: ad2e03)
size/XXL: Extra Extra Large Pull Request (colour: 751f01)

These new labels will help in quickly identifying the scope and complexity of pull requests, facilitating more efficient review processes and project management.
fixes #133

#135  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @JackPlowman and the rest of your teammates on  Graphite

Merge activity

Oct 26, 3:45 PM EDT: A user merged this pull request with Graphite.


Quick fix to issue Posted in #contact-devs

https://discord.com/channels/794206087879852103/848189884614705192/941735116185673770

Just force push it, as I feel a single char isn't really a commit.

Taken into #1173. Thanks, Bailey


Implement a request like library (or port dukcurl)

One big feature yet missing from the legacy duv implementation is the ability to make http(s) client requests. im not sure how nasty the original implementation was and if its better to avoid it but the feature is hardly missing

Http is easy enough given tls.  We'll need openssl bindings or something to get tls though.
Libcurl is an option, but it's tricky to ensure it's non-blocking.

I'm starting on mbedtls bindings for duktape. Initially I'l just choosing the stuff I need for my personal project, but eventually we can add TLS support.
https://github.com/nucleus-js/seaduk/tree/mbed

thats excellent!

do you plan to implement tls basic network functions in the near future?

not near future, sorry.  Life got in the way again.

i wrote a simple request module using the example code given in the test-apps which runs basically fine using request(url,options,callback) unless we have more than one datagram. in this case the simple client example i wrote (attached) fails like this :
Requesting  http://lorempixel.com/800/600/people/
Proto: http://  Host : lorempixel.com  Port : 80 Path : /800/600/people/
Element  Connection  already in headers
DNS lookup of lorempixel.com resolves to 2a01:488:67:1000:253d:c8bc:0:1
Connected to 2a01:488:67:1000:253d:c8bc:0:1:80
ctx: top=1, stack=[{}]
Uncaught TypeError: not buffer
    at [anon] (duk_bi_buffer.c:173) internal
    at concat () native strict preventsyield
    at concat (deps/http-codec:304) strict
    at onRead (deps/wrap-socket:50) strict preventsyield
    at [anon] () native strict preventsyield
    at global (client.js:28) preventsyield
    at [anon] () native strict preventsyield
    at eval (eval:1) preventsyield```

The type of object which it tries to concat is sometimes buffer and sometimes object. Any ideas why the types dont match?

request.zip

module request.js
client example.js


Enhanced queueing for conflicting pessimistic transactions

Background
Currently, we are adopting a simple but rough implementation for pessimistic lock waiting and waking up. It works just fine in most normal cases, however, there will be problems under workloads that has frequent lock conflicts. When there are many requests trying to acquire the lock on a single key in TiKV:

The order that the transactions successfully get the lock has too much randomness
When a pessimistic lock request is woken up after waiting a while in TiKV, it always returns WriteConflict to TiDB so that TiDB will retry the statement.
When many pessimistic lock requests are waiting on the same key in TiKV and the current lock is released, one of the requests will be woken up, and the others will be woken up too after delaying for wake-up-delay-duration (which is a configurable value that defaults to 10ms). If the first awakened transaction's retrying request arrives late, the key might already have been locked by another transaction.

This may lead to some problems:

The latency to acquire the lock might be very unstable, causing high tail latency.
There might be too many statement-retries that are useless but wastes resources.
When the problem is severe, it's also possible to cause errors like lock wait timeout and statement retry limit exceeded.

By carefully adjusting wake-up-delay-duration, we are actually able to try to make it work well in most scenarios. However, it's very hard to adjust it since it's difficult to predict the effect of changing it. It's even harder if a user who don't understand the internal implementation of TiKV want to tune it. Besides, its a store level config, which means, if a user have more than one workloads that has different characteristics in a single cluster, it's impossible to use different wake-up-delay-duration for different workloads.
We expects it would be better to make transactions in such kind of scenarios to execute sequentially, avoiding too much useless statement retries and high tail latency. However, this will be very complicated to implement in our current architecture, especially considering that we needs to guarantee the backward compatibility. We have a complicated design for this problem and it took us much efforts to trying to demonstrate it. We've confirmed that the complicated design is helpful for solving the problem stated above in those kinds of scenarios. However, the stronger queueing behavior in lock waiting introduces higher average latency, or to say, lower QPS. We do have ideas to optimize it further, however, it would make the design much more complicated.
Now we are going to productize the idea. As the first step, we will first implement a subset of the original design: we will enhance the queueing behavior for pessimistic lock requests that involves only one key. This will be helpful for many cases that has frequent lock conflicts on some single rows. And since there's possibly QPS regression, the new behavior will be optional and disabled by default. A user will need to choose to get lower tail latency at the expense of higher average latency if the user needs.

Demo branches
The draft branches for demo can be found at:

https://github.com/pingcap/tidb/pull/35588
https://github.com/tikv/client-go
https://github.com/tikv/tikv/pull/12749

Most part of the final implementation can be ported from these branches, but there will also be some differences.

Tasks

[ ] The RFC / design document

RFC for TiKV: https://github.com/tikv/rfcs/pull/100
Design doc for TiDB: https://github.com/pingcap/tidb/pull/37518

[ ] The spec document

kvproto

[x] Protocol change (kvproto) https://github.com/pingcap/kvproto/pull/872

TiKV

[x] Lock waiting queue implementation

Each key will have its own waiting queue. In the demo branch, it's put in Latch, but later we found hard to store it with an efficient and simple data structure. We will consider store it in the lock table of ConcurrencyManager or another separated concurrent hash map.
https://github.com/tikv/tikv/pull/13486
https://github.com/tikv/tikv/pull/13560

[x] Waiter manager & deadlock detector adaption

WaiterManager no longer have the responsibility to wake up lock requests after the lock is released. Instead, we will need it to be able to index requests by unique tokens. Therefore both its interface and inner data structure need to be updated.
https://github.com/tikv/tikv/pull/13447

[x] New implementation of storage command AcquirePessimisticLock

For requests that contains only one key, it can choose to use the new locking behavior: it's allowed to force lock the key even when there's write conflict. It will use the new protocol to return the actual locked ts (the actual lower bound of for_update_ts) to the client (TiDB).
https://github.com/tikv/tikv/pull/13687

[x] Resuming of pessimistic lock requests

For pessimistic lock requests that choose to use the new locking behavior, after waiting for another lock, we allow it to continue acquiring the lock after being woken up, instead of returning WriteConflict to the client to request a retry.
https://github.com/tikv/tikv/pull/13833
https://github.com/tikv/tikv/pull/13826

[ ] Lock wait info update for queueing requests

When many requests are queued waiting the same key, and one of them is granted the lock after the previous lock being released, then for the other queueing requests, the transaction they are waiting for is changed. In this case, we need some update operations so that it can perform correct deadlock detection and provide diagnostic information about lock waiting.

client-go

[x] Support aggressive locking behavior

https://github.com/tikv/client-go/pull/528

TiDB

[ ] Support aggressive locking behavior which can be enabled by system variable

https://github.com/pingcap/tidb/pull/35588

Misc

https://github.com/tikv/tikv/pull/13865

Optional optimizations

[ ] Region level waking up after region change (leader or epoch change)
[ ] Batch re-detect deadlocks after updating lock wait info
[ ] Replace the implementation of Key from Vec<u8> to something like Arc<Vec<u8>> or Arc<[u8]> to reduce overhead of cloning keys (it needs to be tested whether it helps)

Integration tests

Sorry I just notice that this issue is mistakenly closed when merging https://github.com/tikv/tikv/pull/13865


datasets

Hi，Is there a script that converts the txt file that saves camera poses and camera intrinsics into training cameras.bin and images.bin format? And is the points3D.bin necessary for training?

Please refer to multinerf or DVGO for help on custom settings. We will give more documentations later.


Check for R&D approval to cover all NHIC data

Talk more with JC about confirming we have approvals in place to allow us to upload the information we're collecting for NHIC. Document the outcome for better reference.

JC has a document on this, and has recently sent a copy to MB. See that for reference.


The rotating icon of initializing typescript language server is not centered.

VSCode Version: 1.36.0
OS Version: macOS 10.14.5

If you look at the gif above the two arrow icons that rotate around the center aren't actually set around the center.

Duplicate of #75446


Add structure to CHEBI:63191

Hi Adnan,
Please could you add a structure to CHEBI:63191 (structure with no stereoX  - see  (−)-β-caryophyllene (CHEBI:10357) and (+)-β-caryophyllene (CHEBI:63190) )
Thank you,
Anne

Now added.


some bugfixes in JavaFX threading

init() method is called on Platform thread
Application will close on exit

This change is

Coverage increased (+0.1%) to 65.205% when pulling d9440e832d22b38c6ddef88496e46e734f0e4aaa on javafx-app-threading into 2583502dda7ec6db7deda70d89a312caedbd24ce on master.

There are still some problems. See #181

Reviewed 4 of 4 files at r1.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

platform/dolphin-platform-client/src/main/java/com/canoo/dolphin/client/impl/ClientContextImpl.java, line 83 [r1] (raw file):

        CompletableFuture<ControllerProxy<T>> factoryResult = controllerProxyFactory.create(name);

        return factoryResult.handle((c, e) -> {

variable factoryResult can be inlined

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion.

platform/dolphin-platform-client/src/main/java/com/canoo/dolphin/client/impl/ClientContextImpl.java, line 83 [r1] (raw file):
Previously, aalmiray (Andres Almiray) wrote…
> variable `factoryResult` can be inlined

Done

Comments from Reviewable


react-css-themr should be in peerDependencies

I think the module react-css-themr should be listed in the peerDependencies of the package.json instead of the dependencies. Otherwise, if I use both modules (react-cssthemr and react-toolbox) in a project, I'll end up having the react-css-themr module twice in my bundle.

Is this issue still valid ?

Yes, it's still valid, i still see the dependency, haven't been moved to peer ones yet.


[Bug]: H1 heading to big for smaller screens

hey @schmelto can I try to fix it ?

hey @schmelto can I try to fix it ?

Sure @PritamSarbajna

Hey @schmelto,
If I can change the values inside translateY() then only it'll fix, so is there any way to define a variable inside them and then in any width calling required values ?


[Bug] Edit: Redirect on checkout page to wrong location

          https://github.com/tastyigniter/TastyIgniter/assets/78227663/f9c1be94-e8aa-413c-81be-d4e47319f066

Still got the same problem. i made a video.
Originally posted by @diverds1992 in https://github.com/tastyigniter/TastyIgniter/issues/1064#issuecomment-1880047006

Or is the only option the multi vendor extension?

Yeah, this really needs to be fixed

This issue will be closed and archived in 3 days, as there has been no activity in the last 60 days. If this issue is still relevant or you would like to see it actioned, please respond and we will re-open this issue. If this issue is critical to your business, consider the Dedicated Support Service where a Service Level Agreement is offered.

This issue will be closed and archived in 3 days, as there has been no activity in the last 60 days. If this issue is still relevant or you would like to see it actioned, please respond and we will re-open this issue. If this issue is critical to your business, consider the Dedicated Support Service where a Service Level Agreement is offered.


Feat/limit x y

What this PR does:
Which issue(s) this PR fixes:

Fixes #113
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

Codecov Report

Merging #218 (4dd64b4) into master (a755fb1) will decrease coverage by 0.09%.
The diff coverage is 12.76%.

:exclamation: Current head 4dd64b4 differs from pull request most recent head 9377eae. Consider uploading reports for the commit 9377eae to get more accurate results

@@            Coverage Diff             @@
##           master     #218      +/-   ##
==========================================
- Coverage   36.92%   36.83%   -0.10%
==========================================
  Files          83       84       +1
  Lines       12999    13045      +46
==========================================
+ Hits         4800     4805       +5
- Misses       7596     7636      +40
- Partials      603      604       +1

Impacted Files
Coverage Δ

pkg/runtime/ast/ast.go
78.26% <0.00%> (-0.88%)
:arrow_down:

pkg/runtime/ast/drop_index.go
0.00% <0.00%> (ø)

pkg/runtime/ast/proto.go
57.14% <ø> (ø)

pkg/runtime/optimize/optimizer.go
29.89% <28.57%> (-0.21%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a755fb1...9377eae. Read the comment docs.


Manually closed unsaved queries keep opening with Azure Data Studio

Issue Type: Bug
After closing (manually) unsaved queries, said queries open up again if Azure Data Studio is closed and opened again (even with shutdown of pc in between).
Steps to reproduce:

Close query windows
Close Azure Data Studio
Open Azure Data Studio

Azure Data Studio version: azuredatastudio 1.25.1 (e7eacc32c0d20bd4da843320c3bfac231b308c98, 2020-12-11T01:59:24.271Z)
OS version: Windows_NT x64 10.0.17763

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-7300U CPU @ 2.60GHz (4 x 2712)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: disabled_offopengl: enabled_onprotected_video_decode: enabledrasterization: enabledskia_renderer: disabled_off_okvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
7.88GB (2.07GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions (2)

Extension
Author (truncated)
Version

azdata-sanddance
msr
3.1.0

poor-sql-formatter
WSR
0.1.0

Issue 8792 has a comment which suggests removing:
c:\Users[your username]\AppData\Roaming\azuredatastudio\Backups[random number]\untitled\
Removing these files stopped the reported behaviour.

this keeps on happening -  need to constantly remove the files from \Backups[random number]\untitled\

Still having this exact issue, would be great if there was a fix for this.

I did a quick test with latest build on both Win and Mac (Intel) and I was not able to repro the issue. I'll close the issue for now but feel free to re-open it if you are still seeing this issue and/or have more information for the repro.


Video is always not available on mobile

I'm trying to play some videos with the integrated player but I can't. The app always shows the following screen:

I execute "gradlew assembleDebug" to try the app with my config.

@brapifra Which video encoding you're using?

@MirzaMubasharBaig I've tried H264 and Theora

@brapifra check video block object with what key you have in encoded_videos object of course model (previously there was support for fallback,  mobile_high and mobile_high, but recently fallback  was dropped from Android app it may by cause of the problem with playback)

Where am I supposed to look for this video block? I think I don't have any video block object. I've just upload the video in the unit with Studio

@brapifra this is the exact line of code that you're looking for:
https://github.com/edx/edx-app-android/blob/0fc318a1615839d719d7423820e8787415cc7c56/OpenEdXMobile/src/main/java/org/edx/mobile/view/adapters/CourseUnitPagerAdapter.java#L63-L63

Thans @miankhalid
As @KolomietsOleh said the videos are encoded with fallback, that's the problem. How can I change the encoding to mobile_high or mobile_low?

How can I change the encoding to mobile_high or mobile_low?

@brapifra This is something that requires edX's Studio side knowledge, which unfortunately don't have.
It might be a good idea to join our slack channel for these sort of discussions.
https://openedx.slack.com/

Any updates on this issue?

mobile_high and mobile_low are default profiles set in edx-val. https://github.com/edx/edx-val/blob/c70104e906555b9897a1abea35dc101ad39d8b02/edxval/migrations/0002_data__default_profiles.py#L10
Unless you have edx-val configured, it looks like no video will be available on the mobile app with our current logic. I thought we added a configuration to play all video types. I will dig deeper and report back.

Please have a look at this PR #1061, which should solve the aforementioned issue. Re-open if necessary :)

Hi all,
I got same issue and also check the profile on admin panel

The video is playing on iOS app but on it showing this on Android

Can someone help me with it. Thank you

Check in your app configuration if the USING_VIDEO_PIPELINE flag is enabled or disabled.


Bugfix #18: ReferenceError - process is not defined

Provided a fix for https://github.com/jrgarciadev/react-iconly/issues/18 issue.
This is a critical issue that seems to appear on all projects that are using react-scripts@5 and up.
Tested locally and works as expected. Attached screenshots of the example view and tests.

@jrgarciadev can you please take a look?  😅
Thanks

Btw @ionleu would you like to change microbundle-ctrl by https://tsup.egoist.dev/ ?


Virtual methods in interfaces

Fixes #12418
Fixes #12488
Depends on dotnet/samples#1615
This tutorial explains how you can use default interface methods to declare virtual methods in interfaces. These enable you to create "mixins" that let you declare functionality by declaring which interfaces a type supports.

This caught my eye when skimming over it because it seems to suggest that interface methods by default aren't virtual. However, interface methods are always virtual regardless of having a default implementation or not (and DIMs are always virtual regardless if the virtual keyword is used or not).
So I'm just afraid this may ask learners to ask themselves "so how do I create non-virtual DIMs?".

@dasMulli Thanks for starting this conversation. It sparks some interesting thoughts.
Interface methods are not virtual.  The sections of the spec on Interface implementation inheritance and Interface re-implementation cover the behavior in detail.
That said, I like your suggestion to augment this with how to create non-virtual DIM implementations. I touch on the subject in the tutorial on versioning, but I don't think that's enough.
Where and how can we make this subtlety more clear? How best can we make the distinction between implementing an interface and overriding virtual methods in a base class more clear?

it's complicated. I'm not sure if we're now talking about the interface instance method itself - which is always emitted as .method public hidebysig newslot virtual instance to IL - or the method on the implementing class that the interface is mapped to (which would need to be marked as virtual for overwriting in subclasses if my brain now doesn't fail me completely).
Granted, this is super unintuitive because I can override a DIM in a "sub-interface" without explicitly marking it as virtual, because interface instance methods are always emitted as virtual (but not as abstract)

In addition, because default interface methods can be virtual, the method dispatch always chooses the implementation in the class.

This suggests to me that using the virtual keyword inside the interface definition would change the semantics.

I might as well be missing the key statement here and can't see the wood through all the trees 🙃

@dasMulli  I just ran some tests on the code. I've got some misleading statements in how I explain the concepts here.
I'll update before it's ready for a full review. Thanks for the discussion.

I updated the focus and the content. It's ready for review.  Thanks for the discussion @dasMulli

I'd suggest to also remove the virtual keywords from the interface declarations in the sample code.

How do you feel about removing the virtual declarations from the interfaces in the sample code?

@dasMulli Thanks for the reminder. New PR coming....


fix: pass the encryption key path for the launcher

I changed how the defaults work so I could pass this as a base64 encrypted string. This adds the path to the containers directly.
Verified on both linux and macos.

bors try

bors r+


Release prep for 2.6.0

Base branch for the release.
We are working on the archive of 2.5 and add the release version in the menu for the backend repo.

There are still references to the hugo based version, hence the changes are minimal.


Finalization of Applications

This PR has 3 new pages and database changes.

Impression page - impression to be entered by HR staff.
Finalize page - to finalize the application after all the details have been filled and mark it ready for sending it to HR monitor.
Applications page - Here the not finalized list of applications is listed and also shows the count of applications that are finalized and has send button.

If you are done with the changes suggested, feel free to merge this PR! 👍
The only discussion that will remain open is https://github.com/Nasser-Eddin-Nasser/Digitact/pull/55#discussion_r436714326 but I guess we shouldn't get blocked by this. I still got the opinion that one should not use getRawValue() in the way it was done here.

If you are done with the changes suggested, feel free to merge this PR! 👍
The only discussion that will remain open is #55 (comment) but I guess we shouldn't get blocked by this. I still got the opinion that one should not use getRawValue() in the way it was done here.

Ya Martin, I have a different opinion about getRawValue() still. Anyways I will file an issue regarding it for tracking. I will think of it and will do the needful. Thanks for your suggestions and views on it. It has different perspective than mine.


Resize Terminal Window crash

Setup

Rancher version: 2.7-head (Commit ID: b40b9cda92aa8dc5707485a0e026aa8fb70d2347)
Browser type & version: Chrome, Chromium

Describe the bug

When Terminal Window is resized, there is a critical error. It's necessary to reload the window to continue.
To Reproduce

Go to local cluster homepage.
Click on 'Kubectl Sheel' to open the Terminal.
Nav to a cluster --> Apps --> Charts --> Click on Alerting Driver card (or any other) --> Install --> Next --> View Chart Info
Click on 'Dock to bottom' (the icon on the top-right of Chart Info page)
You should have tabs 'Kubectl' and 'Alerting Drivers' open.
Drag to up the Terminal Window by the resize button.

Result
Message error 'This API only accepts integers' is displayed.
Console output:
console.js:31 Error: This API only accepts integers
at e._verifyIntegers (xterm.js:1:1)
at e.resize (xterm.js:1:1)
at e.fit (xterm-addon-fit.js:1:1)
at VueComponent.fit (ContainerShell.vue:296:1)
at VueComponent.height (ContainerShell.vue:96:1)
at invokeWithErrorHandling (vue.runtime.esm.js:1863:1)
at Watcher.run (vue.runtime.esm.js:4584:1)
at flushSchedulerQueue (vue.runtime.esm.js:4326:1)
at Array. (vue.runtime.esm.js:1989:1)
at flushCallbacks (vue.runtime.esm.js:1915:1)
Expected Result

Any error are expected.
Screenshots

Note - That big 'white page' error only occurs in dev. For production builds the error will be swallowed

During testing on v2.7.0 I was able to reproduce the reported error when following the Steps to Reproduce. As was mentioned by @richard-cox the error was swallowed, but it was visible in the console logs.
However, when testing on v2.7-head (Commit ID: dadb4c0), I did not observe the error when resizing the terminal window either up or down. Therefore I am closing this ticket.
Steps to Reproduce:

Go to local cluster homepage.
Click on 'Kubectl Sheel' to open the Terminal.
Nav to a cluster --> Apps --> Charts --> Click on Alerting Driver card (or any other) --> Install --> Next --> View Chart Info
Click on 'Dock to bottom' (the icon on the top-right of Chart Info page)
You should have tabs 'Kubectl' and 'Alerting Drivers' open.
Drag to up the Terminal Window by the resize button.

v2.7.0:

v2.7-head (Commit ID: dadb4c0):


[Explicit Module Builds] Add libSwiftScan API for scanner cache serialization/deserialization

https://github.com/apple/swift/pull/37723 added API to libSwiftScan to save and restore the dependency scanner's state on the filesystem.
This PR adds this API to the corresponding client code in the driver.

@swift-ci please test

This will need https://github.com/apple/swift/pull/37780 to land in a toolchain snapshot first.

@swift-ci please test


The service is not available now?

I tried several times today, it keeps returning timeout. Any body got the same problem?

I have the same problem. Hope that somebody know the way how it is possibly can be solved (but guess it is highly unlikely).

same prob now , not sure if this can be fixed by changing the src url ? any idea pls

I just tried it and it works fine for me

thanks @Darkblader24 , seems its url issue , could u pls help to using src url for a try , and share the one you were using , such as : translator = Translator(service_urls='translate.google.com')

oh ! i found that fixed by using
translator = Translator(service_urls=['translate.google.cn',])
can't miss that comma !


⚠ Bofrost.de has degraded performance

In 66994f5, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 2176 ms

Resolved: Bofrost.de performance has improved in 10cc52a.


Add Answers.io support

Twitter's Answers now have custom events :)
https://answers.io/answers

Excellent, send us a pull request with it!

Actually I've already implemented this. It was in beta so I hadn't merged the branch yet, but I guess it's publicly available now. I'll merge it this week.
https://github.com/twobitlabs/AnalyticsKit/tree/crashlytics-events

#30


Loading component instead of "Loading ..."

Hello,
This plugin works fine and fits all my needs. I would like to suggest a improvement, instead of the text "Loading ..." it could show a loading component. I think there is no loading component in bootstrap (you know, that circle's one) but with that loading bar would be cool too.
Thanks

I use this script for page:
http://w3lessons.info/2014/01/26/showing-busy-loading-indicator-during-an-ajax-request-using-jquery/
Can be adapted to use the grid.

Thank you, yes it can be adapted.


python: phonenumbers: 8.10.3 -> 8.10.5

Motivation for this change
New release available
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

included in https://github.com/NixOS/nixpkgs/pull/55757


Update tsdoc references

Problem solved
Short description of the bug fixed or feature added

PR-Codex overview
This PR focuses on enhancing the documentation and annotations across various components in the thirdweb codebase, particularly in relation to wallet connections, transactions, and NFT functionalities. It also includes some improvements in the organization of code and examples.
Detailed summary

Updated documentation annotations to include @walletConnection, @transaction, and @nft tags.
Removed the --legacy-peer-deps flag from CLI commands in multiple markdown files.
Introduced new components and improved existing ones in the NFT module (e.g., NFT.Name, NFT.Description, NFT.Media).
Enhanced the getSidebarLinkGroups function for better organization of documentation links.
Added internal utility functions for transaction decorators.
Improved the ConnectEmbed and PayEmbed components with new props and examples.
Refactored various components for better clarity and functionality, including TransactionButton and BuyDirectListingButton.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

#5375
#5374  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @joaquim-verges and the rest of your teammates on  Graphite

Merge activity

Nov 13, 1:53 PM EST: The merge label 'merge-queue' was detected. This PR will be added to the Graphite merge queue once it meets the requirements.


Talk to / implement Bintray

Bintray as a new player could have the same badges as PyPI and other repos (# of downloads, latest version, etc).
The REST API makes this quite easy; the only special thing to take care of is keeping an eye on the rate limits, and cache data so that not every shield request leads to an API request.

It sounds like there's a pretty definitive answer vis-a-vis lifting the rate limit there. Can this move forward then?

What we have for many services is a secret user/token couple. Of course, that's sensitive data that is usually not too hard to re-generate should I... err... accidentally rm the file. Here, it would require extra careful handling, I guess. Beyond that, there's a bit of coding to be done, if anyone is up to the task.

$ curl https://bintray.com/docs/api.html

<title>404 Not Found</title>

404 Not Found

nginx

@PeterDaveHello: The new URL is https://bintray.com/docs/api/ (or https://bintray.com/docs/api/#_get_version for the Get Version method).

Oh yeah, I found it, but it'll be better to redirect instead of 404 ha ...

Why one would CURL to .html?

Just easy to understand and faster then a screenshot.

Got it. Where the old URL comes from?

The top of this issue.

Will add redirect.

+1

This issue, #234, and #284 can all be closed now that #517 has been merged, right?

Implemented by https://github.com/badges/shields/pull/517/files.

Is there way to get version in plain text ( not svg)?

Hi there, feel free to open a new issue with your question.


Error when authorizing user to vm

Installation information

jumpscale version:master(05a7b85)
operating system: Ubuntu

Detailed description
The first time authorization to newly created vm happens it fails. In the second retry form the scheduler it succeeds.
Steps to reproduce
Use a blueprint to create a new VM see:https://github.com/Jumpscale/ays9/blob/master/templates/nodes/node.ovc/readme.md
Relevant stacktraces
Traceback (most recent call last):
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/tmp/actions/node.ovc/4c2f9028ddc45f814c1f850584f1391d.py", line 292, in install
    prefab = _check_ssh_authorization(service, machine)
  File "/tmp/actions/node.ovc/4c2f9028ddc45f814c1f850584f1391d.py", line 393, in _check_ssh_authorization
    prefab = _ssh_authorize(service, vm_info)
  File "/tmp/actions/node.ovc/4c2f9028ddc45f814c1f850584f1391d.py", line 207, in _ssh_authorize
    executor.prefab.ssh.authorize("root", sshkey.model.data.keyPub)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabSSH.py", line 133, in authorize
    d = self.prefab.user.check(user, need_passwd=False)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabUser.py", line 82, in check
    _, d, _ = self.prefab.core.run("getent passwd | egrep '^%s:' ; true" % (name))
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 1108, in run
    cmd = self.replace(cmd)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 275, in replace
    args2 = self.getArgsDict()
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 305, in getArgsDict
    for key, var in self.dir_paths.items():
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 215, in dir_paths
    return self.executor.dir_paths
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorBase.py", line 336, in dir_paths
    self.initEnv()
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorBase.py", line 309, in initEnv
    self.state.configUpdate(TT, False)  # will not overwrite
  File "/opt/code/github/jumpscale/core9/JumpScale9/core/State.py", line 172, in configUpdate
    self.configSave()
  File "/opt/code/github/jumpscale/core9/JumpScale9/core/State.py", line 187, in configSave
    self.executor.file_write(self.configPath, data)
File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorBase.py", line 392, in file_write
    res = self.execute(cmd)
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorSSH.py", line 237, in execute
    rc, out, err = self.sshclient.execute(cmds2, die=die, showout=showout)
  File "/opt/code/github/jumpscale/core9/JumpScale9/clients/ssh/SSHClient.py", line 282, in execute
    "Cannot execute (ssh):\n%s\noutput:\n%serrors:\n%s" % (cmd, out.getvalue(), err.getvalue()))
JumpScale9.errorhandling.JSExceptions.RuntimeError: ERROR: Cannot execute (ssh):
set -e

set -e
mkdir -p /etc
echo "W3JlZGlzXQphZGRyID0gImxvY2FsaG9zdCIKcG9ydCA9IDYzNzkKZW5hYmxlZCA9IGZhbHNlCgpbZW1haWxdCmZyb20gPSAiaW5mb0BpbmN1YmFpZC5jb20iCnNtdHBfcG9ydCA9IDQ0MwpzbXRwX3NlcnZlciA9ICIiCgpbZ3JpZF0KbmlkID0gMApnaWQgPSAwCgpbc3lzdGVtXQpyZWFkb25seSA9I
GZhbHNlCmNvbnRhaW5lciA9IGZhbHNlCmF1dG9waXAgPSBmYWxzZQpkZWJ1ZyA9IHRydWUKClttZV0KbG9naW5uYW1lID0gInNvbWVpZCIKZnVsbG5hbWUgPSAiVGhpcyBpcyBteSBmdWxsIG5hbWUsIG5lZWRzIHRvIGJlIHJlcGxhY2UiCgpbc3NoXQpTU0hLRVlOQU1FID0gInNvbWVuYW1lIgoKW2RpcnNd
ClRFTVBMQVRFRElSID0gIi9vcHQvdmFyL3RlbXBsYXRlcyIKVkFSRElSID0gIi9vcHQvdmFyIgpMSUJESVIgPSAiL29wdC9saWIvIgpUTVBESVIgPSAiL3RtcCIKQkFTRURJUiA9ICIvb3B0IgpCSU5ESVIgPSAiL29wdC9iaW4iCkNGR0RJUiA9ICIvb3B0L2NmZyIKSE9TVERJUiA9ICIvaG9tZS9jbG91ZHN
jYWxlcnMvanM5aG9zdC8iCkxPR0RJUiA9ICIvb3B0L3Zhci9sb2ciCkJBU0VESVJKUyA9ICIvb3B0L2p1bXBzY2FsZTkiCkNPREVESVIgPSAiL29wdC9jb2RlIgpKU0FQUFNESVIgPSAiL29wdC9qdW1wc2NhbGU5L2FwcCIKQlVJTERESVIgPSAiL29wdC92YXIvYnVpbGQiCkRBVEFESVIgPSAiL29wdC92YX
IvZGF0YSIKSE9TVENGR0RJUiA9ICIvaG9tZS9jbG91ZHNjYWxlcnMvanM5aG9zdC9jZmcvIgpIT01FRElSID0gIi9ob21lL2Nsb3Vkc2NhbGVycyIK" | openssl base64 -A -d > /etc/jumpscale9.toml

output:
errors:
bash: line 4: /etc/jumpscale9.toml: Permission denied
 ((type:runtime.error)

Another possible execption:
    result = self.fn(*self.args, **self.kwargs)
  File "/tmp/actions/node.ovc/1c540255f396baec3a6a767b898b4213.py", line 794, in install
    prefab = _check_ssh_authorization(service, machine)
  File "/tmp/actions/node.ovc/1c540255f396baec3a6a767b898b4213.py", line 200, in _check_ssh_authorization
    prefab = _ssh_authorize(service, vm_info)
  File "/tmp/actions/node.ovc/1c540255f396baec3a6a767b898b4213.py", line 217, in _ssh_authorize
    executor.prefab.ssh.authorize("root", sshkey.model.data.keyPub)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabSSH.py", line 133, in authorize
    d = self.prefab.user.check(user, need_passwd=False)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabUser.py", line 82, in check
    _, d, _ = self.prefab.core.run("getent passwd | egrep '^%s:' ; true" % (name))
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 1099, in run
    cmd = self.replace(cmd)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 275, in replace
    args2 = self.getArgsDict()
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 305, in getArgsDict
    for key, var in self.dir_paths.items():
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 215, in dir_paths
    return self.executor.dir_paths
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorBase.py", line 336, in dir_paths
    self.initEnv()
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorBase.py", line 240, in initEnv
    T=self._getDirPathConfig()
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorBase.py", line 168, in _getDirPathConfig
    hostdirexists= self.exists("/host")
File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorBase.py", line 130, in exists
    return self.cache.get("exists:%s"%path, check)
  File "/opt/code/github/jumpscale/core9/JumpScale9/data/cache/Cache.py", line 115, in get
    val = method(**kwargs)
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorBase.py", line 125, in check
    rc,_,_=self.execute('test -e %s'%path,die=False,showout=False)
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorSSH.py", line 237, in execute
    rc, out, err = self.sshclient.execute(cmds2, die=die, showout=showout)
  File "/opt/code/github/jumpscale/core9/JumpScale9/clients/ssh/SSHClient.py", line 214, in execute
    ch = self.transport.open_session()
AttributeError: 'NoneType' object has no attribute 'open_session'

The following error gets triggered when accessing same node.ovc multiple times:
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/tmp/actions/node.ovc/7b1df1cf8d77d04a3f9696a922c647d9.py", line 263, in install
    prefab = _check_ssh_authorization(service, machine)
  File "/tmp/actions/node.ovc/7b1df1cf8d77d04a3f9696a922c647d9.py", line 475, in _check_ssh_authorization
    prefab = _ssh_authorize(service, vm_info)
  File "/tmp/actions/node.ovc/7b1df1cf8d77d04a3f9696a922c647d9.py", line 286, in _ssh_authorize
    executor.prefab.ssh.authorize("root", sshkey.model.data.keyPub)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabSSH.py", line 133, in authorize
    d = self.prefab.user.check(user, need_passwd=False)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabUser.py", line 82, in check
    _, d, _ = self.prefab.core.run("getent passwd | egrep '^%s:' ; true" % (name))
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 1099, in run
    cmd = self.replace(cmd)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 275, in replace
    args2 = self.getArgsDict()
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 307, in getArgsDict
    args["HOSTNAME"] = self.hostname
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 588, in hostname
    return self.cache.get("hostname", get)
  File "/opt/code/github/jumpscale/core9/JumpScale9/data/cache/Cache.py", line 115, in get
    val = method(**kwargs)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 585, in get
    rc, out, err = self.run("cat %s" % hostfile, showout=False, replaceArgs=False)
  File "/opt/code/github/jumpscale/prefab9/JumpScale9Prefab/PrefabCore.py", line 1107, in run
    ppath = self.prefab.bash.profileDefault.pathProfile
File "/opt/code/github/jumpscale/core9/JumpScale9/tools/bash/BashFactory.py", line 284, in profileDefault
    self._profileDefault = self.profileGet(path)
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/bash/BashFactory.py", line 263, in profileGet
    self.executor.file_write(path, "")
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorBase.py", line 392, in file_write
    res = self.execute(cmd)
  File "/opt/code/github/jumpscale/core9/JumpScale9/tools/executor/ExecutorSSH.py", line 237, in execute
    rc, out, err = self.sshclient.execute(cmds2, die=die, showout=showout)
  File "/opt/code/github/jumpscale/core9/JumpScale9/clients/ssh/SSHClient.py", line 282, in execute
    "Cannot execute (ssh):\n%s\noutput:\n%serrors:\n%s" % (cmd, out.getvalue(), err.getvalue()))
JumpScale9.errorhandling.JSExceptions.RuntimeError: ERROR: Cannot execute (ssh):
set -e

set -e
mkdir -p /root
echo "" | openssl base64 -A -d > /root/.bash_profile

output:
errors:
bash: line 4: /root/.bash_profile: Permission denied
 ((type:runtime.error)

Verified


NixOS power.ups module does not generate all needed config files

Describe the bug
The following configuration.nix snippet:
power.ups = {
  enable = true;
  mode = "standalone";
  ups = {
    usb-ups = {
      driver = "usbhid-ups";
      port = "auto";
      description = "A cheapo USB UPS";
    };
  };
};

results in services failing to start due to missing config files etc.
upsmon: Can't open /etc/nut//upsmon.conf: No such file or directory
upsd: stat /etc/nut//upsd.conf: No such file or directory
(upsdrv fails too, but that could be down to the fact that I haven't plugged the UPS in yet -- but for the record, it complains Can't chdir to /var/lib/nut/: No such file or directory)
Expected behavior
I'd expect the services to start successfully (and probably complain about the missing actual UPS).
Notify maintainers
(The UPS module appears to have received only drive-by changes for the last several years, so no idea who to bug personally).
Metadata

system: "x86_64-linux"
host os: Linux 5.4.48, NixOS, 20.03.2351.f8248ab6d9e (Markhor)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.6
channels(root): "nixos-20.03.2351.f8248ab6d9e"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module: services.monitoring.ups

OK, should've read the whole module: I'm supposed to supply the missing /etc/nut files myself because it's unwise to keep them in the store (they could contain passwords and whatnot, I guess?).
Declaring (in the service definitions) the state directory to be /var/lib/nut but creating /var/state/ups for the purpose appears to be a real bug, though :)

Also (sorry for the semi-off-topic) what are the best practices of referencing hand-written config files without their contents winding up in the store?  Double-symlinking?
If this is documented somewhere, then I haven't found where.

I was struggling with the same problem as @cmm until I noticed this bug report.  After copying and editing upsd.conf.sample, upsd.users.sample, and upsmon.conf.sample from /nix/store/...nut-2.7.4/etc/ to /etc/nut/ (and setting chmod 600 on all those files), I was able to get it to work.
I think this should be more discoverable.  If it's not appropriate to copy those sample files to /etc/nut/ on the users behalf, there should at least be detailed instructions on the power.ups.enable option to explain what else needs to be done.
 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.47, NixOS, 20.09pre231796.22a81aa5fc1 (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.6`
 - channels(root): `"nixos-20.09pre231796.22a81aa5fc1"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Agreed this should be more discoverable.  Perhaps at least upsd.conf could be defaulted to empty and written, since I seem to be getting away without anything in there.
In case it helps anyone, I got it working with something like this:
  systemd.services.upsd.preStart = let
    upsd-conf = pkgs.writeText "upsd.conf" "";
  in ''
    mkdir -p $NUT_STATEPATH
    chmod 700 $NUT_STATEPATH
    mkdir -p /etc/nut
    ln -sf ${upsd-conf} /etc/nut/upsd.conf
    ln -sf <some secure file> /etc/nut/upsd.users
  '';

  systemd.services.upsmon.preStart = ''
    ln -sf <some secure file> /etc/nut/upsmon.conf
  '';

I was hit this bug today, too. Maybe we can fix it by using a startup unit that handles the secrets and setup the state directories like nextcloud or syncthing do

Been working on nut today and wanted to put some notes here. Using version 2.7.4.
First, NUT_STATEPATH isn't used by upsd for PIDs and it will try to write to /var/state/ups/. ./configure flags --with-statepath, --with-altpidpath,  --with-pidpath may be needed instead of environment variables until a 2.7.5 release. Ref: https://github.com/networkupstools/nut/pull/473
Using upsd -D for debug makes it non-forking, requiring a change to the systemd unit Type.
I was able to get upsd to run as non-root:
  users.users.nut = {
    isSystemUser = true;
  };

  services.udev.extraRules = ''
    SUBSYSTEM=="usb", ATTRS{idVendor}=="8087", ATTRS{idProduct}=="0024", OWNER="nut" SYMLINK+="usb/ups"
  '';

  systemd.services.upsd.serviceConfig.User = "nut";

I'm still working on understanding how to manage state directories in NixOS but this was needed:
  systemd.tmpfiles.rules = [
    "d '/var/lib/nut/'              0700 nut nogroup - -"
  ];

I set out to get my UPS (cyberpower CP1500 AVR UPS) working with NixOS today. This was the first issue I came across, I figured this was the best place to share my configuration.
I probably made this more complex than it needed to be because I have no idea what I am doing, and it is definitely not secure; one problem at a time.
let
  vid = "0764";
  pid = "0501";
  password = "TODO";
in
{
  # at some point something will make a /var/state/ups directory,
  # chown that to nut:
  # $ sudo chown nut:nut /var/state/ups
  power.ups = {
    enable = true;
    mode = "standalone";
    # debug by calling the driver:
    # $ sudo NUT_CONFPATH=/etc/nut/ usbhid-ups -u nut -D -a cyberpower
    ups.cyberpower = {
      # find your driver here:
      # https://networkupstools.org/docs/man/usbhid-ups.html
      driver = "usbhid-ups";
      description = "CP1500 AVR UPS";
      port = "auto";
      directives = [
        "vendorid = ${vid}"
        "productid = ${pid}"
      ];
      # this option is not valid for usbhid-ups
      maxStartDelay = null;
    };
    maxStartDelay = 10;
  };

  users = {
    users.nut = {
      isSystemUser = true;
      group = "nut";
      # it does not seem to do anything with this directory
      # but something errored without it, so whatever
      home = "/var/lib/nut";
      createHome = true;
    };
    groups.nut = { };
  };

  services.udev.extraRules = ''
    SUBSYSTEM=="usb", ATTRS{idVendor}=="${vid}", ATTRS{idProduct}=="${pid}", MODE="664", GROUP="nut", OWNER="nut"
  '';

  systemd.services.upsd.serviceConfig = {
    User = "nut";
    Group = "nut";
  };

  systemd.services.upsdrv.serviceConfig = {
    User = "nut";
    Group = "nut";
  };

  # reference: https://github.com/networkupstools/nut/tree/master/conf
  environment.etc = {
    # all this file needs to do is exist
    upsdConf = {
      text = "";
      target = "nut/upsd.conf";
      mode = "0440";
      group = "nut";
      user = "nut";
    };
    upsdUsers = {
      # update upsmonConf MONITOR to match
      text = ''
      [upsmon]
        password = ${password}
        upsmon master
      '';
      target = "nut/upsd.users";
      mode = "0440";
      group = "nut";
      user = "nut";
    };
    # RUN_AS_USER is not a default
    # the rest are from the sample
    # grep -v '#' /nix/store/8nciysgqi7kmbibd8v31jrdk93qdan3a-nut-2.7.4/etc/upsmon.conf.sample
    upsmonConf = {
      text = ''
        RUN_AS_USER nut

        MINSUPPLIES 1
        SHUTDOWNCMD "shutdown -h 0"
        POLLFREQ 5
        POLLFREQALERT 5
        HOSTSYNC 15
        DEADTIME 15
        RBWARNTIME 43200
        NOCOMMWARNTIME 300
        FINALDELAY 5
        MONITOR cyberpower@localhost 1 upsmon ${password} master
      '';
      target = "nut/upsmon.conf";
      mode = "0444";
    };
  };
}

Not stale; still relevant

Thanks @newAM for providing your standalone config! I learned a ton tweaking it into a netserver config! The below config creates a working NUT netserver that other machines on your local network can be connected to. The only thing I have yet to figure out is a secure way to include passwords in .nix files that won't show up if I push my local copy to github. Any ideas?
If you want to tweak when NUT shuts down your computers you can modify the values of "override.battery.charge.low = " and "override.battery.runtime.low = " under "directives". SHUTDOWNCMD will be triggered when both conditions are met. The "ignorelb" directive is necessary to have NUT ignore the UPS's defaults.
Anyone using this config may want to make sure SHUTDOWNCMD and NOTIFYCMD point to the correct binaries in your filesystem.
One other thing of note is to check the value of the "schedulerRules" field under power.ups below. You must give it the path to your upssched.conf file. I just put a copy of it inside my /etc/nixos. Could probably add another variable to hold it for convenience.
`
let
vid = "051D";
pid = "0003";
upsname = "apcsmx1500-a";
pass-master = "masterMonitorPassword"; # master password for nut
pass-local = "localMonitorPassword"; # slave/local password for nut
in
{
at some point something will make a /var/state/ups directory,
chown that to nut:
$ sudo chown nut:nut /var/state/ups
power.ups = {
enable = true;
mode = "netserver";
schedulerRules = "/etc/nixos/.config/nut/upssched.conf";
# debug by calling the driver:
# $ sudo NUT_CONFPATH=/etc/nut/ usbhid-ups -u nut -D -a apcsmx1500-a
ups.${upsname} = {
# find your driver here:
# https://networkupstools.org/docs/man/usbhid-ups.html
driver = "usbhid-ups";
shutdownOrder = 0;
description = "APC SMX1500RM2UNC UPS";
port = "auto";
directives = [
"vendorid = ${vid}"
"productid = ${pid}"
"ignorelb"
"override.battery.charge.low = 50"
"override.battery.runtime.low = 1200"
];
# this option is not valid for usbhid-ups
maxStartDelay = null;
};
maxStartDelay = 30;
};
users = {
users.nut = {
isSystemUser = true;
group = "nut";
# it does not seem to do anything with this directory
# but something errored without it, so whatever
home = "/var/lib/nut";
createHome = true;
};
groups.nut = { };
};
services.udev.extraRules = ''
SUBSYSTEM=="usb", ATTRS{idVendor}=="${vid}", ATTRS{idProduct}=="${pid}", MODE="664", GROUP="nut", OWNER="nut"
'';
systemd.services.upsd.serviceConfig = {
#User = "nut";
#Group = "nut";
User = "root";
Group = "nut";
};
systemd.services.upsdrv.serviceConfig = {
#User = "nut";
#Group = "nut";
User = "root";
Group = "nut";
};
reference: https://github.com/networkupstools/nut/tree/master/conf
environment.etc = {
# all this file needs to do is exist
upsdConf = {
text = ''
LISTEN <IP_ADDRESS> 3493
LISTEN <IP_ADDRESS> 3493
'';
target = "nut/upsd.conf";
mode = "0440";
group = "nut";
user = "root";
};
upsdUsers = {
# update upsmonConf MONITOR to match
text = ''
[monmaster]
password = "${pass-master}"
upsmon master
    [monslave]
        password = "${pass-local}"
        upsmon slave
  '';
target = "nut/upsd.users";
mode = "0440";
group = "nut";
user = "root";
};
# RUN_AS_USER is not a default
# the rest are from the sample
# grep -v '#' /nix/store/8nciysgqi7kmbibd8v31jrdk93qdan3a-nut-2.7.4/etc/upsmon.conf.sample
upsmonConf = {
  text = ''
    MONITOR<EMAIL_ADDRESS>1 monmaster "${pass-master}" master

    RUN_AS_USER root

    MINSUPPLIES 1
    SHUTDOWNCMD "/run/current-system/sw/bin/shutdown -h +0"
    NOTIFYCMD /run/current-system/sw/bin/upssched
    POLLFREQ 5
    POLLFREQALERT 5
    HOSTSYNC 15
    DEADTIME 15
    POWERDOWNFLAG /etc/killpower

    NOTIFYMSG ONLINE  "UPS %s on line power"
    NOTIFYMSG ONBATT  "UPS %s on battery"
    NOTIFYMSG LOWBATT "UPS %s battery is low"
    NOTIFYMSG FSD   "UPS %s: forced shutdown in progress"
    NOTIFYMSG COMMOK  "Communications with UPS %s established"
    NOTIFYMSG COMMBAD "Communications with UPS %s lost"
    NOTIFYMSG SHUTDOWN  "Auto logout and shutdown proceeding"
    NOTIFYMSG REPLBATT  "UPS %s battery needs to be replaced"
    NOTIFYMSG NOCOMM  "UPS %s is unavailable"
    NOTIFYMSG NOPARENT  "upsmon parent process died - shutdown impossible"

    NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC
    NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC
    NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC
    NOTIFYFLAG FSD    SYSLOG+WALL+EXEC
    NOTIFYFLAG COMMOK SYSLOG+WALL+EXEC
    NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
    NOTIFYFLAG SHUTDOWN SYSLOG+WALL+EXEC
    NOTIFYFLAG REPLBATT SYSLOG+WALL+EXEC
    NOTIFYFLAG NOCOMM SYSLOG+WALL+EXEC
    NOTIFYFLAG NOPARENT SYSLOG+WALL

    RBWARNTIME 43200

    NOCOMMWARNTIME 300

    FINALDELAY 5
  '';
  target = "nut/upsmon.conf";
  mode = "0444";
  group = "nut";
  user = "root";
};

};
}
`

@justinmakes For secrets in nix config, I think the current recommendation is to put it in a separate file and not include it in the repo. See also https://github.com/NixOS/nixpkgs/issues/24288

I have been using sops-nix to commit encrypted passwords to my NixOS repo, but I have yet to put in the elbow grease to make this work with my upsmon password.

https://github.com/NixOS/nixpkgs/pull/213006 is my attempt at making power.ups generate the config files necessary for basic operation. If you are using power.ups, could you try out the changes and leave a review? Thanks!


Revert: [#1521] No need to reset sample-buffer

Revert commit 1521

Thank you for you contribution!


Dynamic pipeline stages

With the changes introduced with #1865, it is now possible to extend the hook functionality and attach hooks to other hooks. The use case for this becomes trivial with test libraries, where one may want to extend the functionality of a hook from a base class. For example
# File 1
class Base(rfm.RegressionTest):
    ...
    @run_before('compile')
    def set_compiler_flags(self):
        self.build_system.cxxflags = ['-O3']

# File 2
from file1 import Base
class Derived(Base):
    ...
    @run_after('set_compiler_flags')
    def extend_compiler_flags(self):
        self.build_system.cxxflags += ['--amdgpu-target=gfx900']

Won't do


Support DITA composite docs in topicmerge #1077

Signed-off-by: Robert D Anderson<EMAIL_ADDRESS>This fix addresses #1077 as well as an unreported issue I found with a test case for #1077.
Background: if you have a DITA document that uses the <dita> root element, the topicmerge process will only ever preserve the first <topic> in that document. This is true whether you reference the document (href="composite.dita"), the first topic (href="composite.dita#first"), another child of the root element (href="composite.dita#thirdChildOfRootDita"), or a sub-topic anywhere in the document (href="composite.dita#subTopicSomewhere"). This is clearly wrong; no matter what topic you reference, you get the first (and only the first) topic with its children, which may not even include the one you referenced.
Additional background: with a "normal" DITA document that has a root topic: regardless of how a map references the document (just the file, root topic, or sub-topic), you get the whole document in the PDF.
I think there's an argument to be made that if a map references the Nth child of <dita> you should only get the Nth child. But I've based this fix around the current processing for "normal" DITA topics. So: if a map references a topic that starts with <dita>, you get the whole file (this is the normal scenario). If a map references a sub-topic, you still get the whole file (I haven't actually seen this happen but it matches "normal" topics). This is the behavior we've supported internally for 13+ years, and it's the one that's made the most sense for our users.
To make this fix: topicmerge preserves the <dita> element and adds an ID so that maps can reference it. In the XSL post-process, if I reference a <dita> element, I split the reference up - preserving metadata for each reference, but only putting child topics in the end. [I had hoped to mostly handle this in the actual PDF2 code. But it's so heavily based on topic-id keys that any fix would have to add "If parent is <dita> then X else Y" around 20 or 100 times, and would effectively 1) make this a backwards incompatible change and 2) mean nobody with a customization gets the fix.
Unreported issue that's fixed
I realized that if I have a "normal" DITA document with a root topic, but only have the map reference a sub-topic, all links are broken. Basically, the map merge process (lines 214 and 215 ) assume that if you reference a topic ID in the map, that can only be the root topic; so they re-write any references to be to the first topic. That is: if my map has topics.dita#subtopic, lines 214 and 215 ensure that any references to #subtopic are rewritten to go to firstTopicId. This is also the reason that even if a map references the 3rd child of <dita>, you only get the first child of <dita> in your PDF.

Attaching test case map.ditamap with the following conditions:

Reference a topic document
Reference root topic in a topic document
Reference a sub-topic in topic document
Reference a composite <dita> document
Reference first topic in a composite <dita> document
Reference middle topic a composite <dita> document
Reference sub-topic in middle topic of a composite <dita> document
Reference last topic in a composite <dita> document
Reference a composite <dita> document
Include 2 topics that use the same topic IDs
Link to all of those using <xref>; also link to a figure within the referenced sub-topic, and to a figure within the <dita> document that isn't in the first topic (both of these figure links previously failed).

Refactored the code and added test automation. @robander could you review and merge if you're ok with the changes?