Implement columnar encoders and TSM compaction process

Compactions
There are two distinct compaction processes:

TSM compactions, which aim to replace a set of TSM files with versions that are of increased density along with removing any tombstoned data.
WAL Snapshots, which write time / value data from the in-memory cache to level-0 TSM files

For reference, issue #9981 implements the columnar decoders and readers.
For the sake of simplicity, some examples will refer to the Float data type, however, any of the other supported data types (Integer, String, etc) could be substituted.
TSM Compactions
When merging blocks of data for a series + field, the existing compaction implementation decodes and encodes compressed data using the iterative decoder / encoder APIs. This process should be refactored to use batch-oriented APIs to improve performance.
Starting from the top and working down, the key component responsible for merging a set of TSM files is the tsmKeyIterator, which is:

an implementation of the LSM Tree merge operation
that merges a specific set of TSM files
in series-key order
producing compressed and merged blocks of time and value data for each series
in ascending time order
for all series keys across the set of TSM files

Under certain conditions, as an optimization, blocks may not require decoding and re-encoding. This behavior can be observed for Float values:
https://github.com/influxdata/influxdb/blob/a85306c53e2de2fa29c43e951bd9d486bea5dc39/tsdb/engine/tsm1/compact.gen.go#L104-L118
The KeyIterator, defined as:
https://github.com/influxdata/influxdb/blob/1c0e49e00210cd61aa41a1c07e1ca12a3f24595e/tsdb/engine/tsm1/compact.go#L1204-L1221
is an abstraction used by the Compactor to read encoded TSM blocks for each series key. tsmKeyIterator conforms to this interface, such that it can be used by the write function:
https://github.com/influxdata/influxdb/blob/1c0e49e00210cd61aa41a1c07e1ca12a3f24595e/tsdb/engine/tsm1/compact.go#L1128-L1163
to produce a new set of TSM files. Note that iter.Next() is called for each iteration of the loop to move to the next merged block and iter.Read() is called to obtain the data to be encoded. The tsmKeyIterator provides two strategies for compacting data, per the fast option:
https://github.com/influxdata/influxdb/blob/1c0e49e00210cd61aa41a1c07e1ca12a3f24595e/tsdb/engine/tsm1/compact.go#L1240-L1246
The focus for the remainder of this document will be to define the requirements to implement a new KeyIterator, which merges a set of TSM files using the existing batch-oriented decoders and new batch-oriented encoders.
Cache Snapshots
The cache (WAL) is periodically snapshotted to generate a level-0 TSM file. A separate investigation is necessary to understand what would be required to utilize batch encoders for this process.
Tasks
Similar to moving the query / read path to batch-oriented APIs, it makes sense to start from the lowest layers and work up to defining a new KeyIterator.
Unit tests and benchmarks
In order to demonstrate correctness and show the scale of the performance improvements, unit tests should be replicated from the existing encoders. A diary of benchmarks was created to keep track of the improvements to the decoders. It would be useful to maintain a similar diary for the encoders.
Add batch encoders
Similarly to the array decoders, new implementations of the encoders should provide a single API for encoding a block of values. These encoders are responsible for encoding a single column of data, such as timestamps, floats or strings.
An example of a batch decoder in tsdb/engine/tsm1/batch_float.go:
func FloatBatchDecodeAll(b []byte, dst []int64) ([]int64, error)

which decodes the byte slice b, using dst (to potentially avoid allocations) and returning the decoded slice or an error if decoding fails. The Float decoder was rewritten from the iterative approach:
https://github.com/influxdata/influxdb/blob/426a9a0b8b727c1923d83fc5fb7c8eb83ebc18d0/tsdb/engine/tsm1/float.go#L142-L155
Which required the client call Next() bool to decode the next value and Values() float64 to fetch the value. The improvements resulted in a single loop to decode an entire block of values, allowing a host of compiler optimizations.
A similar API for a batch encoder:
func FloatArrayEncodeAll(src []int, b []byte) ([]byte, error)

which encodes the slice src, using the byte slice b and returns the encoded slice or an error if encoding fails.  The definition of the FloatEncoder can be found here:
https://github.com/influxdata/influxdb/blob/426a9a0b8b727c1923d83fc5fb7c8eb83ebc18d0/tsdb/engine/tsm1/float.go#L29-L41
Implement batch encoders

[x] IntegerArrayEncodeAll
[x] UnsignedArrayEncodeAll
[x] FloatArrayEncodeAll
[x] StringArrayEncodeAll
[x] BooleanArrayEncodeAll
[x] TimestampArrayEncodeAll

NOTE: FloatBatchDecodeAll should be renamed to FloatArrayDecodeAll, which follows the Array naming convention established throughout the remainder of the columnar work.
Add batch block encoders
The block encoders are responsible for TSM blocks, encoding a columns of up to 1,000 timestamps and values.
An example of a Float block decoder:
func DecodeFloatArrayBlock(block []byte, a *tsdb.FloatArray) error

decodes the block of time and value data into the FloatArray a using the block decoders. FloatArray, defined as:
https://github.com/influxdata/influxdb/blob/0841c51d9319a8a9fc9f13530759bbf1d0f0c23c/tsdb/arrayvalues.gen.go#L9-L12
maintains the timestamps and values as separate slices. This is knows as a Struct of Arrays layout. FloatArray replaces FloatValues:
type FloatValue struct {
	unixnano int64
	value    float64
}

type FloatValues []FloatValue

which is an array of structs. Struct of Arrays improves cache locality and is better suited to leverage SIMD and is also the layout specified by Apache Arrow.
The tsmKeyIterator uses the FloatValues#Encode API to encode a block of values:
https://github.com/influxdata/influxdb/blob/9cd31520ec2ead0b1975689806a313ebbc092b2f/tsdb/engine/tsm1/encoding.gen.go#L454-L456
which forwards to encodeFloatValuesBlock:
https://github.com/influxdata/influxdb/blob/9cd31520ec2ead0b1975689806a313ebbc092b2f/tsdb/engine/tsm1/encoding.gen.go#L458-L496
The existing implementation uses the iterative encoders and therefore requires a new implementation is required to encode a FloatArray, leveraging the batch encoders:
func encodeFloatArrayBlock(a *tsdb.FloatArray, b []byte) ([]byte, error)

where a is encoded using b and the result returned or an error if there was a problem encoding the data.
Implement batch block encoders

[x] encodeIntegerArrayBlock
[x] encodeUnsignedArrayBlock
[x] encodeFloatArrayBlock
[x] encodeStringArrayBlock
[x] encodeBooleanArrayBlock

Add Encode to FloatArray
Add a FloatValues#Encode method to the the code generation template arrayvalues.gen.go.tmpl
As noted above, the existing FloatValues#Encode method calls the encodeFloatValuesBlock. This task is to add the equivalent FloatArray#Encode methods, calling the new encodeFloatArrayBlock API.
Implement batch block encoders

[x] IntegerArray#Encode
[x] UnsignedArray#Encode
[x] FloatArray#Encode
[x] StringArray#Encode
[x] BooleanArray#Encode

Implement batch-oriented KeyIterator
This task must implement a new version of the tsmKeyIterator using the batch-oriented types, including FloatArray, DecodeFloatArrayBlock and FloatArray#Encode APIs.

[x] tsmKeyIterator replacement

Possible optimizations
It has been observed that large compactions generate a considerable amount of garbage. This is likely due to the merging of blocks:
https://github.com/influxdata/influxdb/blob/a85306c53e2de2fa29c43e951bd9d486bea5dc39/tsdb/engine/tsm1/compact.gen.go#L162
and the later encoding of blocks:
https://github.com/influxdata/influxdb/blob/a85306c53e2de2fa29c43e951bd9d486bea5dc39/tsdb/engine/tsm1/compact.gen.go#L175
and
https://github.com/influxdata/influxdb/blob/a85306c53e2de2fa29c43e951bd9d486bea5dc39/tsdb/engine/tsm1/compact.gen.go#L193
It would be worth investigating ways of recycling memory, as this should also improve performance and reduce GC pressure.

Fixed via #10300.


Fix management context path in URL (#2702)

Codecov Report

Merging #2769 into 1.4.x will increase coverage by 0.08%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##            1.4.x    #2769      +/-   ##
==========================================
+ Coverage   69.34%   69.42%   +0.08%
==========================================
  Files         240      240
  Lines        8472     8475       +3
  Branches     1047     1048       +1
==========================================
+ Hits         5875     5884       +9
+ Misses       2196     2189       -7
- Partials      401      402       +1

Impacted Files
Coverage Δ

...ka/metadata/DefaultManagementMetadataProvider.java
88.63% <100%> (+0.83%)
:arrow_up:

...lix/turbine/stream/TurbineStreamConfiguration.java
86.66% <0%> (+13.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8b92a2...4ac10c5. Read the comment docs.

@darkius is there an issue that this is associated with?  If so can you please include it in the description.  If not please describe the problem.

@ryanjbaxter added issue number

Added a comment. Hopefully it clarifies things


.close() gives exception

Im trying to close my SSE connection but I get an exception.
I initiate like this:
this.source = new EventSourcePolyfill(url, { headers: { "X-Requested-With": "XMLHttpRequest" }  });
And then I close it like this:
this.source.close();
I get the following exception:
Uncaught (in promise) DOMException
And Chrome tells me that the issue is on line 460:
reader.cancel();

I cannot reproduce it, is there a way to setup a server and reproduce it?

Hmm, shame. I can't copy my server side code here.
Do you have any idea why it would cause an exception in this case? It's really ruining my error log with all the errors.
Can I "swallow" the exception in some way?

I also get this when calling close() using version 1.0.5 and Chrome 74.0.3729.169 on Mac
Uncaught (in promise) DOMException

(anonymous) | @ | eventsource.js:460
-- | -- | --
  | close | @ | eventsource.js:809
  | EventSourcePolyfill.close | @ | eventsource.js:888
  | es.onmessage | @ | sse-notifications-receiver:151
  | fire | @ | eventsource.js:602
  | onProgress | @ | eventsource.js:758
  | (anonymous) | @ | eventsource.js:470

The connection is closed, it just that it causes an exception.
Seems like a client or browser issue, if this is a server issue, is there a sample server that works with this client such that close() does not cause an exception?
What server was used for the development and testing of this client?

@Yaffle I am able to reproduce the issue using the example server code in the Readme.
I created a Node server using the example server js and used the index.html also as per the example.
I had to modify the index.html to explicitly create an instance of EventSourcePolyfill instead of just EventSource as recent versions of chrome already include EventSource so the polyfill is not used. If the polyfill is not used, the close() exception does not occur.
I just added a setTimeout to close the connection after 5 seconds and boom, the Dom exception occurs.
Here is my modified index.html (i also modified the script path /src/eventsource.js in both the index.html and server.js to match my file structure)
<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <title>EventSource example</title>
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <script src="/src/eventsource.js"></script>
    <script>
      // use EventSourcePolyfill explicitly
      var es = new EventSourcePolyfill("events.php");
      var listener = function (event) {
        var div = document.createElement("div");
        var type = event.type;
        div.appendChild(document.createTextNode(type + ": " + (type === "message" ? event.data : es.url)));
        document.body.appendChild(div);
      };
      es.addEventListener("open", listener);
      es.addEventListener("message", listener);
      es.addEventListener("error", listener);
      es.addEventListener("close", listener);

      window.setTimeout(function(){ es.close(); }, 5000);
    </script>
</head>
<body>
</body>
</html>

ok, I was able to reproduce it, thank you.
It seems,

It is better not to call both of controller.abort() and reader.cancel():

        controller.abort();
        reader.cancel();

It is needed to not reject the error if it is an AbortError.

Version 1.0.7 works great, thanks!

Thank you for solving the issue! Greatly appreciated!

For anyone interested, AbortError is still happening in Firefox, but imo it is now a Firefox issue.
StackBlitz: https://stackblitz.com/edit/firefox-fetch-abort-error
Firefox bug report: https://bugzilla.mozilla.org/show_bug.cgi?id=1583815

@earshinov ,
IMHO, it can be reproduced like this:
function fetchSomething() {
  var controller = new AbortController();
  var signal = controller.signal;
  fetch('data:text/plain,test', {signal}).then(function(response) {
    const reader = response.body.getReader();
    controller.abort();
    //reader.cancel(); - this needs to be used to avoid the log message, seems
  }).catch(function(e) {
    console.log('catched', e);
  }).finally(function(e) {
    console.log('done');
  });
}

fetchSomething();

@earshinov, a workaround was added, but it is still throws on the page unload

@Yaffle ,  I guess all errors can be fixed by subscribing to the Promise returned from Reader.cancel:
reader.cancel().catch(() => {});

After this change I cannot reproduce the AbortError in Firefox any longer.
I updated my StackBlitz example: https://stackblitz.com/edit/firefox-fetch-abort-error


Add __init__.py to projects.

Add __init__.py to project directories so that project modules can be imported like:
import scenic
from scenic.projects import ncr

Thanks @eringrant for sending the PR.
I've set up automatic adding of __init__.py when exporting our code to the github and  #478 takes care of it.
If you don't mind, we would like to drop this PR as manually adding init is no long needed.

Sounds good; thanks for adding the __init__.pys!


Add support for POSIX Cap in CMake build

This adds a new CMake option that is ON by default: ENABLE_POSIX_CAP. If POSIX cap is not found, the configure step will fail and the user must disable the feature explicitly.

[approve ci centos]

Rebased on master to resolve conflicts.


Speed ocr

I using tessract Android (tess-two) on nexus 7 and pc(core i5), with 1 image 1024x200
nexus 7:  10s and pc: 1s, what happen? Is there any way improve speed ocr when using in Android? Please help me! Thank.

Please use tesseract forum for asking support - see FAQ.

Some things that might make it faster are:
Select a smaller region from mGray where your text is, before createBitmap - so the more heavy methods that follow process a smaller image.
Changing Bitmap.Config.ARGB_8888 to Bitmap.Config.RGB_565 - your image is grayscale, it will not need a ARGB bitmap.
more may help: OCR technology: improve OCR speed


Update libgit2 to 0.27.0-rc1

Also update libssh2 to 1.8.0 (we were at 1.5.*).

I think I am 😆

@Uncommon @tiennou I am extremely busy right now. Can someone create a release draft in this repo following the last releases as an example?

Draft ready. I tried to stick with the old style but stopped short of thanking myself like a 💯 times 🤣.
I'll submit a WIP 0.27 PR while we wait for libgit2-0.27 to 🚢. Also, I hope the iOS linking is okay, as I can't help with that 😟.


Various fixes to workflow dropdown G-UI Components

💬 Description
There's a few here so I'll explain them in the comments.
✅ Checklist

[ ] Tests written
[ ] Browser tested
[ ] Added to documentation
[ ] Added to storybook

 Workflow Mob Test


Fix Siteminder logoff URL for Production

During the logoff process for Siteminder authenticated portals (Institution/Ministry), the logoff URL must be configured appropriately to ensure that the logoff happens on the Siteminder side as well (that it is session based).
Production is currently configured with the same configuration from DEV/TEST environment and needs to be adjusted.

[ ] Update GitHub production environment variable SITE_MINDER_LOGOUT_URL to use https://logon7.gov.bc.ca/clp-cgi/logoff.cgi (confirmed by @JasonCTang ).
[ ] Update the API POD on production to start using the new config.
[ ] Test the logout for Ministry and Institution to ensure that the below error is no longer happening and the new production URL is used instead of the "https://logontest7.gov.bc.ca".

@JasonCTang updated the GitHub config and applied the change to the production SIMS-API POD.
The URL is not using the PROD Siteminder address as expected.


Ignore e2e in coverage

Signed-off-by: bryanl<EMAIL_ADDRESS>

Pull Request Test Coverage Report for Build 890

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+1.8%) to 66.798%

Totals

Change from base Build 889:
1.8%

Covered Lines:
8822

Relevant Lines:
13207

💛  - Coveralls


Current page detection & links of translations for multilingual mode

Small fixes I've done for my personal purposes. Maybe you will like it too.

Small fix for current single page detection by absolute URL comparing (inspired by ".IsMenuCurrent returns false for single pages" thread).
NB: it seems that user should add / to the end of 'url' menu entry param for correct absolute URLs comparing (url = "/projects/") in config:

[[menu.main]]
identifier = "projects"
name = "projects"
weight = 20
url = "/projects/"

Added multilingual mode switcher in the footer (footer.html, before "copyright") with list of links for existing translations of current page in the form of shell commands like menu entries (example output). In config:

[languages]
[languages.ru]
    LanguageName = "русский"

[languages.en]
    LanguageName = "english"

Added default charset (UTF-8) in 'head.html'.

This is great, a very belated thank you @bedefaced!


[🐞] no secondary navbar on docs on mobile

Which component is affected?
Qwik City (routing)
Describe the bug
Now there's no way to get to the second menu navbar to navigate with on mobile devices
Reproduction
https://qwik.builder.io/docs/overview/
Steps to reproduce
open link and use mobile emulation
System Info
not relevant

Additional Information

on it 👍

thanks a lot @zanettin ! 🙏


plotly_marker_change

change poorly marker to new version.
test code:
import cfr
print(cfr.__version__)
pdb = cfr.ProxyDatabase().fetch('PAGES2kv2')
fig = pdb.plotly_count()
fig.show()
fig1 = pdb.plotly()
fig1.show()

Thanks for the fix!
.plotly() works now but .plotly_count() still returns glitches.


Issue Closed and Marked as "Ready To Merge", but these states are invalid.

Hello! #16544 was closed and marked as "Ready To Merge", but I don't think any work was done toward resolving it. I can at least be sure that no work was done that referenced the issue itself.
In the discussion on the issue, it was acknowledged as a bug (at least that's how I understood this comment), but one which may be difficult to resolve.
We have done a lot of work to get around the issue, but it still persists, and still causes us some problems occasionally. I'm making this new issue because I would hate for the discussion on #16544 to be lost and for the original concern never to be addressed.

Yeah, thank you for the report, we will fix it soon


What linux distro are you using?

Hello ! I discovered your post on r/unixporn and I really love this theme and I was wonderinf if you can share some info about your linux distro, font config,hinting, antialising anything that made that text to look so good, I have been trying for ages to acomplish that.
If you can, please share your fontconfig file

OS: Arch Linux
Terminal Emulator: Tilix
Font: FuraCode Nerd Font Mono

I don't have any font config yet, here are my dotfiles.
For status line configuration, I wrote a gist to explain it: gist

So how is the text rendering so sharp in your linux distro ? Is that the default behaviour?

Oh, that's because I'm using surface device, the dpi is very high.

Thank you !

Can you share the name of your surface machine and some info about it?

Surface Pro 5
To be honest, in my mind, the dpi of any surface device is very high.

That's good, I will check them out in the future


Drop support for EOL Python 3.4

Includes #79, the last commit https://github.com/xolox/python-coloredlogs/commit/036faef9fde9421aabb01d6e7f48d34c6bbdc359 here is new.
Here's the pip installs for coloredlogs from PyPI for January 2020:

category
percent
downloads

3.7
43.45%
247,468

3.6
34.84%
198,447

null
6.02%
34,265

2.7
5.87%
33,427

3.5
4.91%
27,946

3.8
4.48%
25,516

3.4
0.44%
2,506

2.6
0.00%
7

3.9
0.00%
7

Total

569,589

Source: pip install -U pypistats && pypistats python_minor coloredlogs --last-month

Coverage remained the same at 92.013% when pulling 036faef9fde9421aabb01d6e7f48d34c6bbdc359 on hugovk:rm-3.4 into b63fce3fbf112c568ef89e1670d511a98f64df80 on xolox:master.

Thanks Hugo, this has been merged and released as part of coloredlogs 12.0.

 I'm impressed by the application of release early, release often 👍🏼


Tokotron: Tokenized TTS for the SpeechBrain benchmark

This pull request contains a modified version of Tokotron, a basic sequence-to-sequence TTS

@flexthink  could I close this PR? since we have another one containing all new updates?


NVS Partition Generator incorrect or esp32 incorrect reading pre build nvs (IDFGH-6821)

Environment

Development Kit:     ESP32-D
Module or chip used:  ESP32-WROOM-32
IDF version: v4.4
Build System:  idf.py
Operating System:   Windows 10
Using an IDE?: Yes Visual Studio with VisualMicro
Power Supply:    USB

Problem Description
I prepared NVS partition using manual from https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_partition_gen.html There was no errors. But after I uploaded partition to the esp32 board I saw that all keys exists but all values are empty.
Output log
NVS key 'batch', type '33', len=-1515870811, val=
NVS key 'otaSalt', type '33', len=-1515870811, val=
NVS key 'authSalt', type '33', len=-1515870811, val=
NVS key 'actKeySalt', type '33', len=-1515870811, val=
NVS key 'actKeyPart', type '33', len=-1515870811, val=
NVS key 'test', type '20', val=-1515870811

Code to show nvs data
esp_err_t err;
const char nvsName[] = "pwds";

err = nvs_flash_init_partition(nvsName);
printf("\nnvs init pwds err=%d\n", err);

nvs_handle_t nvsHandle;
err = nvs_open("storage", NVS_READWRITE, &nvsHandle);
printf("\nnvs open pwds err=%d\n", err);

nvs_iterator_t it = nvs_entry_find(nvsName, "testPwds", NVS_TYPE_ANY);
while (it != NULL) {
nvs_entry_info_t info;
nvs_entry_info(it, &info);
it = nvs_entry_next(it);

if (info.type != 33) {
	int v;
	nvs_get_i32(nvsHandle, info.key, &v);
	printf("NVS key '%s', type '%d', val=%d\n", info.key, info.type, v);
}
else {
	char buf[100];
	size_t len;
	err = nvs_get_str(nvsHandle, info.key, buf, &len);
	printf("NVS key '%s', type '%d', len=%d, val=%s\n", info.key, info.type, (int)len, buf);
}
};

partitions
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x1E0000,
app1,     app,  ota_1,   0x1F0000,0x1E0000,
nvs_key,  data, nvs_keys,0x3D0000,0x1000
pwds,     data, nvs,     0x3D1000,0x3000
spiffs,   data, spiffs,  0x3D4000,0x2C000,

NVS csv file text
key,type,encoding,value
testPwds,namespace,,
batch,data,string,batch111
otaSalt,data,string,11111111111111111111111
authSalt,data,string,2222222222222222222222222
actKeySalt,data,string,3333333333333333333333333
actKeyPart,data,string,44444444444444444444444
test,data,i32,12324241
How it was done

prepared it with command:

PS C:\Users\igord\Documents\esp32-experiments\nvs> python C:/Users/igord/esp/esp-idf/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py generate .\nvs-pwds.csv nvs-pwds.bin 0x3000
C:\Users\igord\esp\esp-idf\components\nvs_flash\nvs_partition_generator\nvs_partition_gen.py:28: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils.dir_util
Creating NVS binary with version: V2 - Multipage Blob Support Enabled
Created NVS binary: ===> C:\Users\igord\Documents\esp32-experiments\nvs\nvs-pwds.bin

It looks ok.
2. Then I uploaded it:
esptool.py -p COM4 -b 921600 --before default_reset --after no_reset --chip esp32 write_flash --flash_mode dio --flash_size detect --flash_freq 40m 0x3D1000 nvs-pwds.bin
esptool.py v3.2
Serial port COM4
Connecting...........
Chip is ESP32-D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 3c:61:05:30:70:1c
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x003d1000 to 0x003d3fff...
Compressed 12288 bytes to 444...
Wrote 12288 bytes (444 compressed) at 0x003d1000 in 0.2 seconds (effective 601.6 kbit/s)...
Hash of data verified.

Leaving...
Staying in bootloader.

Strange that it creates keys but all data alway empty. It's writable but I need pre uploading nvs data with all values. There is no any advice in documentation and I think it must not work so)

@igor-d-n Please check the return values of the function calls to IDF APIs. The you'll probably see a ESP_ERR_NVS_NOT_FOUND, because you use the handle associated with namespace "storage" to find entries of namespace "testPwds".
See the lines in your code vs. in the NVS csv file:
err = nvs_open("storage", NVS_READWRITE, &nvsHandle);

testPwds,namespace,,

@0xjakob thx, I'm blind=) Th was other mistake. If you use not default partition you must use function
nvs_open_from_partition(nvsName, nvsNamespace, NVS_READONLY, &nvsHandle);

@igor-d-n No problem. But there are two things here:

NVS partition names: As you stated, if you're not using the default partition, you have to use different functions for initializing and opening handles where you specify the partition name.
namespaces: You have to use the correct namespace when searching, writing, reading entries. Entries with the same name but different namespace are treated as distinct entries.


Upgrade govuk frontend and enable link styles

The new link styles are currently opt-in. The main differences are:

Links now have underlines that are consistently thinner and a bit further away from the link text.
Links also have a clearer hover state, where the underline gets thicker to make the link stand out to users.

Version 3.13.0 of govuk-frontend disables ink skipping of underlines in hover state to avoid large gaps appearing in the underline on hover in Safari and Firefox.

Mr @adamsilver says ‘Yes’!


Update cloudknox-howto-create-approve-privilege-request.md

Minor updates to make the doc consistent to the current UX.

@marcelodiiorio : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Use NSTemporaryDirectory helper function instead of hard coding /tmp

NSTemporaryDirectory properly checks the environment variable TMP.

@swift-ci please test and merge


Cleaning up

Things to cleanup / not API related

[x] remove subset from trajectories. This is not used and the API has changed a lot. I propose to replace this by a utility function that created a subset snapshot
[x] remove/simplify select_legal_sample in PathMovers. This has also changed a lot and we do not need much of this
[x] remove clustering.py this is very old and not maintained. It will reappear for MSMs but not in the current form
[x] as discussed with @dwhswenson we should write a test_tools_something (put sane name here) that will provide all typically used testing helpers when running from test_tools_something import * (moved to #590)
[ ] fix scripts

with #533 merged, I marked "remove clustering.py" as done

@dwhswenson  and me agreed to remove scripts for 1.0 and leave that as a feature for future updates. There are lots of examples to show how you can do the same and we have not used the scripts in months.


webui: Example Cash Payments over $10k (FinCEN Form 8300)

On our WASM demo site let's have a page for creating cash_payments forms in the browser. It would show off our library and help users out to learn the library (and generate forms).
https://github.com/moov-io/fincen/blob/main/cash_payments/activity.go

https://github.com/moov-io/fincen/pull/21


Update spin version to 5.0

Also remove pointer_method feature which is now stable

Thanks! Could replace the 0.5.* wildcard with 0.5.0 for consistency? Cargo interprets 0.5.0 as ^0.5.0, so it is equivalent to the wildcard here (and wildcards are generally discouraged because they also allows cargo to select very early versions).

Thanks! Could replace the 0.5.* wildcard with 0.5.0 for consistency? Cargo interprets 0.5.0 as ^0.5.0, so it is equivalent to the wildcard here (and wildcards are generally discouraged because they also allows cargo to select very early versions).

Yes makes sense, updated the PR. Thanks

Thanks!

Released as version 0.6.4


Add proposed additions for IS-07 transport support

Adds the following:

Support for MQTT and WebSocket transport
Indication of the transport type for each ID via /transporttype (for where IS-04 is unavailable)
Support for externally defined parameters which are not core to a given transport type

These will require review from the wider community.
Note that in order to avoid git merge/rebase issues, these commits are intentionally developed on top of https://github.com/AMWA-TV/nmos-device-connection-management/pull/50 and should not be merged until that PR is merged.

Hi Andrew,
Can you explain what the ext_subscription_data constraint is for the WebSocket transport?

Hi Cristian. It's purely an example to show how the ext_ parameters could be used, not necessarily how any specific existing implementation does use them.

Ok understood.


Is there any functionality in rasa_core that chatbot can learn it-self based on the past conversion (Statefull Chatbot)? & Is rasa_core stores the chat conversion any where?

Rasa Core version:
Python version:
Operating system (windows, osx, ...):
Issue:

I am developing State full chatbot which learns it-self based on the past conversion done, like which questions are asked most what questions are out of the scope. So Rasa_core provide similar kind of functionality?
For the conversion analysis where the conversion chat is going to store?

Is there any one who help me out from this?


Extract test code from each applet to a separate test.py file

While the "one file per applet" paradigm was nice for a while, we have outgrown it, and the new directory structure reflects that already. Having tests in the main applet file breaks test discovery with just about any Python test framework. We should have tests in the glasgow/applet/<category>/<name>/test.py, next to glasgow/applet/<category>/<name>/__init__.py which contains the applet itself.

Note that this will need some additional work to unbreak glasgow test, which I will do once the mechanical part of this issue (the conversion to multi-file structure) is done.


drop install_crowbar_init

the crowbar-core and crowbar-init package are now responsible of the symbolic links

I think we can also drop the want_postgresql variable.

I think we can also drop the want_postgresql variable.

yeah probably, but it is still used in the old upgrade CI

@MaximilianMeister Right, but it is anyway broken and we will not fix it. I would suggest to create a new PR for this and let's discuss there.

Gating passed critical part already.

 (3) S50P13 #14 Create/Connect Crowbar DB: Service Integration


AC: MTCNN support

Jenkins please retry a build

@IRDonch , @vladimir-dudnik  could you please take a look and merge? It helps to start this model validation in light-weight accuracy and preparation on OpenVINO validation team side.
The changes do not affect current validation flow because MTCNN evaluation delivered as separated script.


Cherrypick 1.4.2

For an explanation of how to fill out the fields, please see the relevant section
in PULL_REQUESTS.md
Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
[Optional Fixes #Issue]
[Optional Deprecated:]

DO NOT press merge.


can't launch binary

Performed 'manual installation' (copy Hyperland, hyperctl, libwlroots.so.11032, .conf and .desktop) but can't launch hyprland on openSUSE Leap 15.4:
Hyprland: error while loading shared libraries: libwlroots.so.11032: cannot open shared object file: No such file or directory

well you didnt copy / copied to the wrong place libwlroots.so.11032, no?

To /usr/lib like described.
..copy the wlroots .so (wlroots.so.XX032) to /usr/lib/.

maybe try /usr/local/lib/

Now I'm getting:
Hyprland: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by Hyprland)
Hyprland: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by Hyprland)
Hyprland: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by Hyprland)
Hyprland: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/libwlroots.so.11032)
Hyprland: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /usr/lib/libwlroots.so.11032)

Hmm
openSUSE comes with glibc- 2.31.

unlucky then. Means suse's glibc is pretty ooold.
iirc, the SUSE maintainer has mentioned it only works on tumbleweed.

No, that is why you don't build binaries against arch and claim that it works on any distro

ok mr big smarty, hyprland requires gcc from may this year. I am pretty sure Void has gcc and stdlib++ older than this, so it's not unreasonable to assume SUSE has too.

Your point is?

So there's nothing I can do than wait. Anyway, thank you for the support.
 Keep on the great work.

you can compile gcc 12.1 yourself, technically.

Think I'll wait :-)

I copied libwlroots.so.11032 to both /usr/lib and /usr/local/lib but I am still getting the same error of cannot open shared object file: No such file or directory.
I am on Fedora 37 btw.

cannot open what so file? Fedora has some outdated deps.

I am getting this exact error: Hyprland: error while loading shared libraries: libwlroots.so.11032: cannot open shared object file: No such file or directory

make sure the so is in ldpath

the $LD_LIBRARY_PATH was unset so I ran sudo ldconfig and then I got another error saying that libseat.so.0 was missing but that was fixed by running sudo dnf install libseat. Now I have another error saying libxcb-errors.so.0 is missing or not found. I cannot find that package anywhere in the fedora repos or anything similar to it. I tried building from source but I could not build and am getting obscure errors when running automake.

see the fedora discussion.

Yeah saw that but my dumbass forgot to run sudo before building. Now when running the Hyprland command getting this in red text: [types/wlr_drm_lease_v1.c:713] No DRM backend supplied, failed to create wlr_drm_lease_v1_manager. I think libdrm is outdated, or is there some way to fix this. Thanks for your help, learned a lot about linux.

no clue

Did you complier libxcb-errors.so.0 file successfully? If you will share with me, that should be much thank for you, Because I don't want to build it from source !!


fix: generate source map if lines are prepended

This corrects the line offset when debugging a module that has lines injected by the importAnalysisPlugin (ie: import.meta.env or import.meta.hot)

The injected code are all in one line and should not cause line offsets, can you provide a repro?

The injected code are all in one line and should not cause line offsets, can you provide a repro?

Ah my bad, I didn't look close enough. Never tested it, so it's probably fine.

Ah my bad, I didn't look close enough. Never tested it, so it's probably fine.

@yyx990803 Looks like CJS imports do cause line offsets, breaking source maps.


Remove install_usercode_dependencies from java instance main

installUserCodeDependencies is only valid for python instance; and python and java have different behaviors on
handling command args. so remove it from java instance to avoid confusion and problems

run java8 tests


Remove the description of dropping 'NS' prefix

Remove " We will also drop the 'NS' prefix from all Foundation classes." according to https://github.com/apple/swift-evolution/commit/bf2e4c4bcb1ad3dd072920f33514546e75cf59bb

We can discuss the goals of the project on the mailing list.

Hi! In my understand, Swift 3 are not going to drop the NS prefix. README.md was written before this change. Therefore, this sentence is no longer valid and should be removed. It will confuse people.

It is still part of the plan, we are iterating on exactly how to do it.

Thanks for this information @parkera - do you have links to the discussion/iteration? I think there's a lot of interest in how the Foundation APIs evolve.

Not just yet, but soon.

I think this proposal is one step in removing "NS" prefixes: https://github.com/apple/swift-evolution/blob/master/proposals/0069-swift-mutability-for-foundation.md


Save Class Probabilities during Inference

During inference prediction, the results.csv outputs the filename, optional offset (sliding window), prediction, and output (list of logits) for each sample.
This should be extended to include the output probabilities as well.

classification: include a separate column for each class in the dataset
regression: include a separate column for each target (re-scaled): this requires the target transform to have names for the target similar to classification. We currently do not yet support multi-target regression but this may be good to add then as well.

I anyways wanted to add multi-target regression, so that's a nice addition anyway (would need to adapt the encoder a bit, I presume).
To achieve this, I think we need to extend _predict to also return something like this: {key: float(output[0, index].numpy()) for index, key in enumerate(self.target_transform.labels)}
However, this depends on #40

During inference we have the following pipeline (M = Model, T = target transform): M.outputs -> T.predict_batch -> T.decode (map) -> (decoded).
During output tracking we currently call  M.outputs -> T.predict_batch -> T.decode (map) -> (save dataframe).
We could extend both pipelines to include e.g.  M.outputs -> T.probabilities_batch -> T.predict_batch -> ... with T.probabilities_batch applying the following operations:

(single-target) regression: identity OP; still scaled (no update of T.predict_batch required)
multi-class classification: Softmax (no update of T.predict_batch required; we only argmax the outputs)
multi-label classification: Sigmoid (remove Sigmoid from T.predict_batch)

Then in all cases we can save the outputs as well as the result of T.probabilities_batch (+ destructure the probabilities to separate columns in the inference results.csv).

To achieve this, I think we need to extend _predict to also return something like this: {key: float(output[0, index].numpy()) for index, key in enumerate(self.target_transform.labels)}

We then can adapt the target transforms to destructure outputs and probabilities as well, simplifying inference here,.
For  (multi-target) regression this would require saving the name of the target column (probably makes most sense).

Yes, that's the plan I had in mind as well! Do you have time to prepare a PR?

Yes i will try to implement it soon.


[ffi] missing type argument in @Native crashes the compiler

@Native() // should have been @Native<Void Function()>
external void foo();

crashes the compiler
type 'DynamicType' is not a subtype of type 'FunctionType' in type cast

#0      FfiNativeTransformer.visitProcedure (package:vm/transformations/ffi/native.dart:671:58)
#1      Procedure.accept (package:kernel/ast.dart:3590:40)
#2      Transformer.transform (package:kernel/visitor.dart:1801:21)
#3      Transformer.transformList (package:kernel/visitor.dart:1836:18)
#4      Library.transformChildren (package:kernel/ast.dart:614:7)
#5      Transformer.defaultTreeNode (package:kernel/visitor.dart:1858:10)
#6      TreeVisitor.visitLibrary (package:kernel/visitor.dart:513:35)
#7      FfiTransformer.visitLibrary (package:vm/transformations/ffi/common.dart:530:26)
#8      FfiNativeTransformer.visitLibrary (package:vm/transformations/ffi/native.dart:95:26)
#9      List.forEach (dart:core-patch/growable_array.dart:416:8)
#10     transformLibraries (package:vm/transformations/ffi/native.dart:41:13)
#11     VmTarget.performModularTransformationsOnLibraries (package:vm/target/vm.dart:179:26)
#12     KernelTarget.runBuildTransformations (package:front_end/src/fasta/kernel/kernel_target.dart:1568:19)
#13     KernelTarget.buildComponent.<anonymous closure> (package:front_end/src/fasta/kernel/kernel_target.dart:638:7)
<asynchronous suspension>
#14     withCrashReporting (package:front_end/src/fasta/crash.dart:136:12)
<asynchronous suspension>
#15     IncrementalCompiler.computeDelta.<anonymous closure> (package:front_end/src/fasta/incremental_compiler.dart:413:23)
<asynchronous suspension>
#16     computeKernel (package:frontend_server/compute_kernel.dart:434:9)
<asynchronous suspension>
#17     main (file:///b/s/w/ir/x/w/sdk/utils/bazel/kernel_worker.dart:32:18)
<asynchronous suspension>
Target //experimental/users/iinozemtsev/flutter_ffi/java/com/google/android/apps/flutterffi:flutterffi failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 4.323s, Forge stats: 0/1 actions cached, 0.8s CPU used, 0.0s queue time, 0.0 MB ObjFS output (novel bytes: 0.0 MB), 0.0 MB local output, Critical Path: 2.07s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
ERROR: Build did NOT complete successfully
INFO: Streaming build results to: [http://sponge2/61bf32f9-abf1-4f5c-ac21-ffc11828d43b](https://sponge2.corp.google.com/61bf32f9-abf1-4f5c-ac21-ffc11828d43b)
iblaze ERROR 12:35:48 blaze build failed: exit status 1

i want to work on this

@faizan003 be warned that next time you will go across the repo posting meaningless ChatGPT (or similar) generated answers you will get banned from organization.


Update starter template to use WaitFor & add application tracing source

This updates the starter template to use the new WaitFor feature and register tracing for a source named for the application.
Contributes to #6076
Microsoft Reviewers: Open in CodeFlow

Do we want to add/update any tests to check that the changes work as expected in the generated projects?

Would it make sense to follow the same convention in the playground apps, or other test apps?

Do we want to add/update any tests to check that the changes work as expected in the generated projects?

I'm not sure what we'd add honestly. As long as we test WaitFor and we have tests that cover that starter works in general (i.e. the projects startup and load), we should be fine I think.


FST plotting function error

The FST function should give the standard error or Z score when called on the diagonal. However the first row is swapped with the first column. Please see the figures below. The first is the plot with just FST and the second is a plot which should show Z score on the diagonal.

Hi @KellyLBennett, just having a look at this now, would you be able to post the function call you made to generate the plots above so I can try to replicate?

Hi alimanfoo I believe this was when running the following
fst_df= ag3.pairwise_average_fst(region="3L:15,000,000-41,000,000", cohorts=wild_cohorts,min_cohort_size=10,site_mask="arab")
ag3.plot_pairwise_average_fst(fst_df,annotation ="Z score")

Thanks @KellyLBennett. Can you provide the value of the wild_cohorts parameter?

Sorry yes, here it is.
wild_cohorts = {
                    "Mwea_2007" : f"taxon == 'arabiensis' and location == 'Mwea' and year == 2007 and insecticide == 'Untested' and partner_sample_id not in {misplaced_samples_list}",
                    "Mwea_2014" : f"taxon == 'arabiensis' and location == 'Mwea' and year == 2014 and insecticide == 'Untested' and partner_sample_id not in {misplaced_samples_list}",
                    "Teso_2013" : f"taxon == 'arabiensis' and location == 'Teso' and insecticide == 'Untested' and partner_sample_id not in {misplaced_samples_list}",
                    "Turkana_2006" : f"taxon == 'arabiensis' and location == 'Turkana' and insecticide == 'Untested' and partner_sample_id not in {misplaced_samples_list}",
                    "Kilifi_2012" : "taxon == 'arabiensis' and location == 'Kilifi' and year == 2012"}

Thanks @KellyLBennett, I can replicate with this slightly simplified example:


Can't compile project when Blackbone is included

Hey, it gives me bunch of errors when i include blackbone and compile my project.
What i do wrong? Thank you for any reply.
Errors

Included Dirs:
Blackbone\src
Blackbone\contrib
Included Libs:
Blackbone\Release
Instruments v141, SDK 10.0.16299
My Code
#include "Injector.h"
#include "BlackBone/Process/Process.h"

using namespace blackbone;

void InjectorClass::Inject(MemoryStruct data)
{
	Process thisProc;
	std::vector<DWORD> dwFoundProc = thisProc.EnumByName(L"blob.exe");
	if (dwFoundProc.empty())
		return;

	thisProc.Attach(dwFoundProc[0]);
	const auto image = thisProc.mmap().MapImage(data.size, static_cast<void*>(data.memory), false, RebaseProcess | NoDelayLoad);
}

VS 17.7 is required to compile it starting from commit 5b8d077939355089d5c07e662b6e36dfb28aa6fa

@DarthTon switched to 17.7, same issue

Looks like you don't link with blackbone lib. Either specify it in Linker project options or add a bllackbone project to your solution and add it as a reference.

@DarthTon tryed all, same issues. This happens only when i include "BlackBone/Process/Process.h"

No, that's just a dependency. You need this

@DarthTon Thanks, now all fine.


fix role endpoint description

fixes role endpoint descp. in all versions, as well as persistent typos

The Permissions sections need to be reverted too, please see comment above.


Add 'foldr1' to the 'Foldable1' typeclass

foldr1 here should behave similar to foldr from the standard prelude, right?

foldr1 from standard Prelude has type (a -> a -> a) -> t a -> a.
The idea is that if it's  nonempty structure, you don't need the "starting value" for the fold.
foldr1 from relude has type (a -> b -> b) -> b -> f a -> b.  Note the extra parameter. Is this difference in type intended?

@jgm Removing the default value would be nice indeed. However, it doesn't look trivial to do. I see, that even semigroupoids doesn't provide foldr1, and I don't see from Hoogle any library that provides safe foldr1 with the nice type signature.

Yeah, no base case is simple when:
-- base
foldr  :: (a -> a -> a) -> t a -> a

But remove the base case for:
-- base
foldr  :: (a -> b -> b) -> b -> t a -> b
-- relude
foldr1 :: (a -> b -> b) -> b -> t a -> b

Is completely another thing.
AFAIK the base cases should not be removed. Everyone expects fold to have a function and then a base case. Base cases are free in provisioning and in the code performance. And giving base cases is a good FP practice. Base cases are essential for FP programming, Monoid empty, Nothing, bool, maybe, either, fromMaybe and etc. all give base cases first.
When to try to remove the base case too hard, the base foldr1 is what comes out as a result, a one-sided non-general fold. and then people have this sort of discussion. To say like in the kindergarten: we see that the base: foldr1 is one being weird here.

I guess that's my question exactly.  Why not have the signature
Foldable1 t => (a -> a -> a) -> t a -> a

Foldable1 is for nonempty structures; we have a way to get head and last, so why do we need to specify a base case?  This would make foldr1 a drop-in replacement for Prelude's, more or less, instead of something completely different.  If you need the flexibility of
(a -> b -> b) -> b -> t a -> b

and don't mind specifying a base case, just use foldr which already looks like this.  Obviously there's something I'm missing.

Because fold for non-empty structures if to take-out the base case would be in an imaginary world we imagine is:
-- imaginary
(a -> b -> b) -> t a -> b
-- base
(a -> a -> a) -> t a -> a
-- relude
(a -> b -> b) -> b -> t a -> b

If to pick real one - it is the relude one. Because the (a -> a -> a) -> t a -> a is not practical.
It is much more practical to just give base case that is for free and have proper catamorphism.
Then to reduce what was for free to have a partial solution that constricts the implementation.
We actually participate in a years long epic discussion on why that foldr1 was put there, it is a holly war, I used to read about it years ago, even before I learned Haskell properly.

I don't mean to engage in a holy war, but what confuses me is this.
NonEmpty is already an instance of Foldable, which has
foldr :: (a -> b -> b) -> b -> t a -> b

So what exactly is the difference between foldr and foldr1 (with relude's type)?

That *1 naming used consistently. foldr1 is Foldable1, which gives what Foldable1 does - gives a guarantee that the structure is empty-safe. While foldr with Foldable does not gives that guarantee and allows non-total unsafe functions.

While foldr with Foldable does not provide that guarantee and allows non-total unsafe functions.

foldr from the Foldable class is perfectly safe for empty structures, isn't it?  It's not a partial function.  Maybe I'm missing something, but I think you must have meant to be talking about foldr1 from Foldable.  Yes, that function is not safe for empty structures.  So I would have expected Foldable1 to provide a safe replacement foldr1 that is restricted to nonempty structures but otherwise works like foldr1 from Foldable (and has the same signature).  Instead, relude offers a function that has the same signature as foldr from Foldable, and (if I understand correctly) is equivalent to it (within a more limited domain of application) and thus redundant.  Wouldn't it be more useful to provide a safe foldr1 with the original signature?  That would allow you to do something you can't already do with foldr.  I don't see how it helps to give us a function that just does what foldr already does but with a different name.
foldr1 (with the original signature) is useful when there's no default base case, but you want to start with first element in the structure, whatever it happens to be.  I've seen cases like this; to achieve safety in these cases, I've just used foldr and a Maybe, but that's a bit awkward and I would have rather had a safe foldr1 (with no base case) that is restricted to nonempty structures.

Since foldr does not have NonEmpty boundary, one can use (a -> b -> b) on empty t a that is unsafe, the most simple "on the lap" example is to evaluate strictly -  \ xs acc -> acc <> head xsor for that matter\ xs acc -> f (acc <> head xs)`, or any function that implicitly uses root of the structure or the tail of it.

(within a more limited domain of application) and thus redundant.

This almost does not happen in Haskell, especially by thoughtful authors. In Haskell for "more limited domain of application" people often use words "specialized to", "more powerful", "optimized for". As if used properly when needed, special cases optimize better, for some cases MonadPlus for parsers is more powerful then Alternative, Abelian structure is much more performant then Semigroup, more restrictions, more properties - more implementation optimization possible. The weaker structures in most cases can be used, but frequently they work less efficiently. The best the most understandable example that I've seen is comparing Associative to Associative & Commutative, what is only Associative is strictly one-threaded sequential, the Commutative gives complete parallelism.

I would have rather had a safe foldr1 (with no base case) that is restricted to nonempty structures.

There are none such. NonEmpty structures always have the base case, that is what to do for NonEmpty value.
So if one (a -> a -> a) -> t a -> a one also needs to provide the base case, but now one needs to do it inside of the (a -> a -> a), which is a limiting signature.
relude: foldr1 allows the same thing, except that it does not require to pack base case into (a -> a -> a) requirements, the base case for t a now can be anything b, and catamorphism can be anything: (a -> b -> b) -> b.
Hence, reludes version is super to bases, and former called consistently, while the (a -> a -> a) -> t a -> a is a limiting fold that works only for some cases, it is still handy and useful, but needs other name.

Since foldr is not NonEmpty typesafe, on empty t a the unsafe (a -> b -> b) can run on it.

Sorry, I'm just not following.  Can you give an example where
foldr :: Foldable t => (a -> b -> b) -> b -> t a -> b

from Foldable blows up on an empty structure (t a)?  It's a totally defined function, is it not?  In what sense, exactly, is it unsafe to use on empty structures?


Updated grid.dart

Updated per https://github.com/flutter/flutter/pull/1938 feedback

lgtm, but perhaps you want the person who left you the feedback to review?

Code LGTM, but I don't really understand the spacing+padding model now.


How to use 'dexie-observable' and 'dexie-relationships' addons?

I am using Dexie.js in my ionic2 project and I am trying to add 'dexie-observable' and 'dexie-relationships' together in one Database, like so:
import Dexie from 'dexie';
import 'dexie-observable';
import relationships from 'dexie-relationships';
...

export class DataBase extends Dexie {

    public matches: Dexie.Table<MatchModel, number>;

    constructor() {
      super("UserDB", { addons: [relationships] });
...

the problem is  'dexie-relationships' overrides the 'dexie-observable' addon, is it possible to use both addons in the same database?
i tried also this:
import Dexie from 'dexie';
import dexieObservable from 'dexie-observable';
import relationships from 'dexie-relationships';
...

export class DataBase extends Dexie {

    public matches: Dexie.Table<MatchModel, number>;

    constructor() {
      super("UserDB", { addons: [relationships, dexieObservable] });
...

or
...
import Dexie from 'dexie';
import dexieObservable from 'dexie-observable';
import relationships from 'dexie-relationships';

Dexie.addons.push(dexieObservable);
...

used versions:
"dexie": "2.0.0-beta.11"
"dexie-observable": "1.0.0-beta.3"
"dexie-relationships": "1.2.8"

solved with #568 pull request


want to update Bibframe vocab but it was hand edited?

The bibframe vocab is out of date;  however, there is a commit 15664793 about adding a property by hand to the existing file.   I'm afraid if I follow the instructions using the rake task update vocabs, it will cause more problems than it solves, especially as the hasAccompaniment property is NOT in the bibframe vocab at http://bibframe.org/vocab-list/.
Plus #26 implies that the rake task will create some additional errors.
What is the best way forward?

I went to re-load this, to look at the differences, and it seems that we are already using the latest version of this vocabulary.
The process is to run the rake task: rake lib/rdf/vocab/bibframe.rdf and then inspect the differences to see if they are expected.
Please close if this satisfies your issue.

yes - it was my bad.  bibframe is a moving target and I was wrong about which thing I was looking at was up to date.


How to Restrict future dates in this framework?

Hi,
I want restrict user to does not allow future dates and also restrict user to below than current date>
Is is possible in this framework.

@lveeramani ,
I will add this feature in a future version of the date alias.
For now have a look at https://github.com/RobinHerbots/jquery.inputmask#postvalidation


Question: Handle arrays properly

Hi,
what is the best way to expose items of arrays?
store/index.js
export const state  = () => {
  foo: [
    { bar_1: { attr: 'change me' } }
  , { bar_2: { attr: 'change me 2' } }
  ]
}

export const mutations = {
  do_something (state, { key, value }) {
    // how to handle this properly???
  }

page/foo.vue
<template>
  <input v-model = 'bar_1.attr'/>
  <input v-model = 'bar_2.attr'/>
</template>

<script>
export default {
  computed: {
    ...takeState('foo')
    .expose( ??? )
    .commit('do_soemthing')
    .map()
  }
}
</script>

I would like to expose all attr attributes of the items, namespaced by bar_1 and bar_2. Or at least expose bar_1 and bar_2. Could you help me please with this and show (a) possible solution(s)?
Would be great. But I don't want to handle this one by one, instead I would like to pass and handle the  array at once and get all the getters and setters for each item in the array.

Hi, in such situation i create component, to which item via props is passed, using v-for.
and, after that, you can expose your item using take() (not takeState)
Foo.vue
<template>
  <bar-edit :bar="bar" v-for="bar in foo" />
</template>

<script>
export default {
  computed: {
    ...mapState('foo');
  }
}
</script>

BarEdit.vue
<template>
  <input v-model = 'attr'/>
</template>

<script>
export default {
  props: ['bar']
  computed: {
    ...take('bar')
    .expose( [ 'attr' ] )
    .commit('do_something')
    .map()
  }
}
</script>

Thank you for your quick response!
Looks like a reasonable approach. Thank you for that and please keep going with this good work :+1:
I'll use it know in my projects :smile:

Just one more question. It seems to be vue specific.
I want to pass all attributes as props, but it seems that in computed propertys this is not available?!. Can you tell me what I'm missing?
export default {
  props: {
    item: {
      type: Object
    , required: true
    }
  , attribute: {
      type: Array
    , default: () => ['value']
    }
  , commit: {
      type: String
    }
  }
, computed: {
    ...take(this.item)
    .expose(this.attribute)
    .commit(this.commit)
    .map()
  }
}

This is not working, but I don't get why. o0

Solved some parts. Who is able to read, has a clear advantage. (german saying) ^^
In the docu about vuex is written:
// in full builds helpers are exposed as Vuex.mapState
import { mapState } from 'vuex'

export default {
  // ...
  computed: mapState({
    // arrow functions can make the code very succinct!
    count: state => state.count,

    // passing the string value 'count' is same as `state => state.count`
    countAlias: 'count',

    // to access local state with `this`, a normal function must be used
    countPlusLocalState (state) {
      return state.count + this.localCount
    }
  })
}

https://vuex.vuejs.org/guide/state.html#the-mapstate-helper
If you want to use this you need to use a function...
I did this an it worked:


Additional information for Table 4

Hey,
regarding Table 4 in your paper. Is it possible for you to share how VOS performed on the different datasets and not only share the average across them? I trained your model locally and would like to validate that I have not done anything wrong during the training process. Because my FPR95 on SVHN is significanly larger than your reported average value (with DenseNet).
Thank you in advance!

Hi, @danielkorth , you cannot compare the average results with a single result on svhn. For the reported result on svhn, please follow the instructions in the readme file and load the model with the provided pretrained models.


Are you planning to release RxDataSources.framework.zip?

Hi.
We are using RxSwift. Thank you very much.
I have one question.
Are you planning to release RxDataSources.framework.zip?
With that, the time for carthage update is shortened, I am very happy.

Hi @shindyu ,
It should build really fast if you only do --platform ios. So far we are only planning on attaching RxSwift.framework.zip

@kzaher
Thank you for the advice! I will try!


golang-protobuf allows to use the bytes type as string

Is your feature request related to a problem? Please describe.
Now bytes is []byte in golang it does not support equal operation, it cannot be used as key in map.
Describe the solution you'd like
Optionally map bytes to string in golang
Describe alternatives you've considered
I tried to use string to replace bytes in proto file, but since protobuf's string is explicitly defined to use UTF8 text, it will generate an error when using non-UTF8 binary text.
Additional context
Envisaged solution:
First, in the proto file, you can enable option go_bytes_as_string = true;.
The corresponding field in the go file generated by protoc is of type string.
Therefore, it is the goal of bytes type to implement equal operator and as the key of map in golang.
There are the following comments in the golang code.
// string is the set of all strings of 8-bit bytes, conventionally but not
// necessarily representing UTF-8-encoded text. A string may be empty, but
// not nil. Values of string type are immutable.
type string string

Does this mean that it is perfectly possible to treat string as a ByteString-like implementation.

So, this proposal I think has come up before https://github.com/golang/protobuf/issues/922
Unfortunately, for a proto file wide option of option go_bytes_as_string = true; we would really want this buy in to happen at the master protobuf side, rather than Golang implementors unilaterally adding in a new potentially sometimes incompatible initiative.
While sort-of frustrating, you can always convert a []byte into a string with an explicit cast in order to use it as a hash key. I don’t know just how feasible that would be though.

@puellanivis got it right — this would need to be addressed at the protobuf level, not at the protobuf go level: https://github.com/protocolbuffers/protobuf


Unit tests

We need unit tests for the JSON <-> MD conversion

Any thought about how those would be written? Using jest or more traditional mocha-style tests?

Roundtrip tests would be useful. To answer my own question, jest seems like a fine choice. Here is a small example:
test('can roundtrip basic text', () => {
  const markdownText = 'This is some text.';
  const value = fromMarkdown.convert(markdownText);
  const markdownRound = toMarkdown.convert(value);
  expect(markdownRound).toEqual(markdownText);
});

Note that even that first test fails round-tripping:
  ● can roundtrip basic text

    expect(received).toEqual(expected) // deep equality

    Expected: "This is some text."
    Received: "This is some text.

    "

This may tie into some of the whitespace issues mentioned in https://github.com/accordproject/cicero-ui/issues/43

Hacktoberfest:
This issue can provide for multiple pull requests.

@irmerk I would like to work on it. Does the tests should be only on the JSON <-> MD conversion? What all scenarios should we consider for the testing, like all the buttons in the toolbar?

@codingdefined this would be great to pick up! Just keep aware of the changes we're making in PRs #114 and #117 over the next few days.
We're using the markdown-transform now, so the conversions are described more in that README. I think the more tests the better, essentially. Testing buttons, testing copy-pasting, testing adding a clause template... If you'd use the editor in X way, we should test X.

@irmerk Created the pull request for the same.

As mentioned in #134, this issue can provide for many PRs, so I will open this up.

Please note the conversation in markdown-transform #117 for testing this repo.

Closing for https://github.com/accordproject/web-components/issues/14


improve spi traits

rename to SPI traits to possibly better names

SpiAll -> HalSpi
SpiEnabled -> HalEnabledSpi
SpiDisabled -> HalDisabledSpi

reset no longer needs to take ownership of the given SPI peripheral

The rename seems fine, thanks! I agree that the reset method should use a &mut self receiver, but your implementation seems a bit extreme. My suggestion would be instead:

Move the implementation from the trait to the ordinary API, so that the reset method is available even when not using the traits
Add a private method like fn internal_disable(&mut self) that contains the register writes, and use that method in both the pub fn disable(self) and pub fn reset(&mut self) methods. If you like, private methods can be implemented for both typestates (inside impl<EN> Spi<$SPIX, EN, $TY> { ... }) whilst keeping the external typestate API.

bors r+


team-building tab bar blue badge

@keybase/react-hackers @keybase/picnicsquad
Desktop

iOS

Android

@cecileboucheron, do you want to review the visuals?

Lgtm!


Package
lucia
Description
Is it possible to realize it? If yes, can you please add an example?

Yes,

After creating a session, send the session ID back in the response body
Store the session ID in device
Send the session ID within the request (e.g. authorization header)

https://lucia-auth.com/guides/validate-bearer-tokens


Event on('licenses') shows empty licenses

So the on Licenses event doesn't really trigger when it actually has fetched the licenses.
I hacked around the problem by adding a setTimeout like the example below, but it of course isn't neat.
Result => Null
steamUser.on('licenses', function(){
    console.log('Result', steamUser['licenses']);
});

Result => Array[] of Licenses
steamUser.on('licenses', function(){
    setTimeout(function(){
        console.log('Result', steamUser['licenses']);
    }, 0); // setTimeout 0ms actually lasts 13ms
});

This is not a bug. As per the documentation, the licenses property will be updated after the event is emitted.
The first property of the event's callback function is the new array of licenses. The reason for this is because the event is emitted when your owned licenses changes, and having both the old and new lists available allows you to compare them.
What you should be doing is this:
steamUser.on('licenses', function(licenses) {
    console.log('Result', licenses);
});

Ah my bad, overlooked the callback with the licenses argument.


build fixed?

Fixes #38

@rultor merge it

@rultor merge it

@amihaiemil OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge it

@amihaiemil Done! FYI, the full log is here (took me 3min)


Difference Between Current Code and Original Paper

Learning rate decay. In the original paper, the learning rate decay starts from 0.001 and is reduced to 0.0005, 0.0003, and 0.0001 after 500K, 1M and 2M global steps respectively. While the code uses a fixed learning rate of 0.001.

no batch-norm for conv1d in encoder (https://github.com/Kyubyong/tacotron/issues/12).

wrong size of conv1d in CBHG, Post-processing net (https://github.com/Kyubyong/tacotron/issues/13)

CBHG structure in post-processing net does not use residual connection. This may be a compromise,  because the residuals are added only if the dimensions are the same. The original paper is unclear.

The last layer of decoder uses a fully connected layer to predict the mel spectrogram. The paper says that it is an important trick to predict r frames at each decoder step. It is unclear whether T = T' or T! = T' in the process of [N, T, C] -> [N, T ', C * r]. The code keeps T=T', but it is also possible that  T' = T / r with frame reduction.

Decoder input problem. The paper says, in inference, only the last frame of the r predictions is fed into the decoder (except for the last step). However,  the code uses all of the r frames. During training, there are the same problem, every r-th ground truth frame is fed into the decoder, rather than all of the r frames.

What about the pre-emphasis 0.97?


implement pay sui

the main changes in this PR:

add paySui transaction type including its transaction construction
change existing pay(), instead of always making new coins and assigning the new coins to recipients, the updated pay() tries to reuse the existing input coins, updating them to proper amounts and assigning to recipients.

the main reason that we want to re-use the input coin, is that gas payment coin needs to live through the transaction, meaning that either coins to be deleted in one transaction or coins created in a transaction cannot be the gas payment coin of this transaction.

also added some unit tests

@sblackshear  sorry for the confusion, my original idea was that, we do not have to withdraw from each coin, instead we can withdraw from the total_coin as long as we make sure at the end, all coins under recipients have pre-defined amounts && the left coins under sender total value would match, but as this is obv. super critical, I tried to minimize the changes but still the change was sort of big.
I also updated & added several tests, suggestions and comments are more than welcome!

I had a chat with @lxfind and the main takeaway is that, for the "transfer all Sui" use-case, "transfer all Sui strictly following the specified amounts" and "using input coin as gas payment" are conflicting, bc we do not know the exact gas cost of the transaction (gas estimation can give a number but it is not exact).
so about use case, we can transfer all coins to one recipient:

without specifying the amount, we make one new coin with balance that is equal to the total balance of input coins
transfer this coin to the recipient and deduct the cost from the coin
we can do so via similar logic as what transferSui does today.

alternatively we can specify the amounts and use pay(), where the transferred amounts are exact, but we will need a separate gas coin.
any wisdom? @sblackshear  if that makes sense, I will revise the PR accordingly.


Loading spinner positioned at center of the page

https://github.com/NewtonFernandis/IssueHunter/assets/103115789/5865f347-7c20-4878-a36d-2413594d80e8

Plus, Made title bold so it stands out from other typography

Thank you soo much for your contribution 🚀 ❤


Main.storyboard: error: The document “(null)” requires Xcode 8.0 or later

xctool version : 0.3.1
Command: script -q /dev/null xctool -configuration Release -workspace

xctool doesn't support building anymore.


Cannot set valAxisMinVal to 0

The if condition in line 3355 of pptxgen.js would return false if opts.valAxisMinVal is 0 and therefore it won't be set.

Thanks @GiridharGNair !


Update jackson-module-scala to 2.9.9

Updates com.fasterxml.jackson.module:jackson-module-scala from 2.9.8 to 2.9.9.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention @scala-steward in the comments below.
Have a nice day!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [{ groupId = "com.fasterxml.jackson.module", artifactId = "jackson-module-scala" }]

labels: semver-patch

Codecov Report

Merging #56 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #56   +/-   ##
=======================================
  Coverage   98.47%   98.47%
=======================================
  Files           8        8
  Lines         131      131
=======================================
  Hits          129      129
  Misses          2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00718c2...7be7476. Read the comment docs.


Expose message attribute on ActionSheetIOS

Hello,
I noticed with the upgrade of ActionSheetIOS to UIAlertController, we now have the ability to pass a message prop... currently though it's set to nil in RCTActionSheetManager.m. Is there a reason to not expose message? if not, would you accept a PR to implement this? I believe it would only work in iOS 8+ (according to the UIActionSheet class reference).
Version: 0.18.0
Thanks!

I don't think there's any reason not to, you can submit a PR to add the prop. Maybe just mention in the doc that it only works on iOS 8+.

That has been already fixed in https://github.com/facebook/react-native/commit/928fd0d60512566b4047f709431ee565499e506f#diff-b6ec6c60921763a2cc180dd8ed24ba35


Episode idea: install kubernetes the hard way on bare metal

install kubernetes the hard way, component by component (pki, etcd, kube-apiserver, kube-scheduler, kube-controllermanager, kubelet, kube-proxy, cri, cni, ...) on bare metal (or virtual machines) with metallb as loadbalancer.

There exist already similar demonstrations (except metallb) e.g.

https://linuxacademy.com/course/kubernetes-the-hard-way/
https://www.youtube.com/watch?v=NvQY5tuxALY


http: periodic HCM access logs

Commit Message: http: periodic HCM access logs
Additional Description: Make it possible to log periodically from the HCM. This parallels the work already done in TcpProxy.
Risk Level: low
Testing: unit tests.
Docs Changes: none
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

/assign-from https://github.com/orgs/envoyproxy/teams/first-pass-reviewers

/assign-from @envoyproxy/first-pass-reviewers

Please resolve the conflict first, then we can merge this patch. Thanks. 😄


Clicking on an item in a dropdown should close the dropdown

When clicking the dropdown menu arrow on a need or connection and then clicking e.g. on the rdf button, the menu stays open.

not necessarily needed -> afaik there are only actions that result in a view change (e.g closeConnection or gotopost) which "close" the contextmenu as is, the  rdf links open the rdf content in a different tab or toggle the rdf in the post-messages, we might want to close the contextmenu there


Calculating mean using filtered data to avoid NaNs

In the presence of NaNs in reference data, program crashes even after using RemoveNaNDataPointsFilter since the mean is being calculated on non-filtered data. Also addresses the issue #67.

Great, looks good. Thanks for looking into that!


Go To Definition doesn't show me the actuall implementation (VSCode)

Type: Bug
When I press F12 or use the Go To Definition option, it only shows me the definition without any implementation. Similarly, when I click on Go To Implementation, it doesn't work either, it shows me No Implementation Found For "BlaBla"
Btw I was using C# Release version it was work fine tho, but when I upgraded to the pre-release version I got this problem

It works fine on my own code, but when I'm trying to use it on an external library it will show me just the definition without any implementation.

Sample :
public class User : ClassFromExternalLibrary
{
   // code here
}

When I press "F12/go-to-def" on ClassFromExternalLibrary  will show me the definition with this on top of the file:
#region Assembly ExternalLibarayNameHere, Version=2.0, Culture=neutral, PublicKeyToken=null
// /home/user/.nuget/packages/ExternalLibarayNameHere/2.0/lib/net6.0/ExternalLibarayNameHere.dll
#endregion

I'm using :

VSCode Information :
Version: 1.80.2
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 6.4.4-200.fc38.x86_64

C# extension (v2.0.312 "Pre-Release version")

C# Dev Kit extension (v0.3.18 Pre-Release version)

Dotnet version (8.0.100-preview.6.23330.14)

Everything is work fine (autocomplete, intellicode, go-to-def(on my own code))

Extension version: 2.0.312
VS Code version: Code 1.80.2 (2ccd690cbff1569e4a83d7c43d45101f817401dc, 2023-07-27T20:41:52.615Z)
OS version: Linux x64 6.4.4-200.fc38.x86_64
Modes:

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
2e4cg342:30602488
f6dab269:30613381
a9j8j154:30646983
showlangstatbar:30737416
vsctsb:30748421
03d35959:30757346
pythonfmttext:30731395
pythoncmv:30756943
fixshowwlkth:30771522
showindicator:30785052
pythongtdpath:30769146
i26e3531:30792625
gsofa:30797621
pythonnosmt12:30797651
pythonnoceb:30797650
e537b577:30795824
dsvsc013:30795093
dsvsc014:30797589

any updates ? @dibarbet

Exactly, I was trying to to see the implementation/code of any external function/class like a function from Nuget package
Btw I solved the issue by disabled C# DevKit extension, I'm not sure if it was an issue with ```
C# DebKit

But I do need to use this extension `C# DevKit`

Dupe of: https://github.com/dotnet/vscode-csharp/issues/5969

@CyrusNajmabadi the dupe links to this issue.  Meant to link another one?

This should be in this weeks release (Wednesday).

Same problem here,Was the lateset version of C# DevKit settled?

It does work, but you need what we described here https://twitter.com/ilspy/status/1697115935629267198
Enable dotnet.navigation.navigateToDecompiledSource

NEED HELP Unable to decompile and view the classes of interfaces in the assembly.
Name: C# Dev Kit ID: ms-dotnettools.csdevkit Version: 0.5.2

It works for me on this version v2.0.436

What if I wanted to view the source of packages using the package symbols?
I find this very odd because when I ctrl+click something, it will take me to the decompiled code but when I step into the code from the debugger (which uses the symbols), it gives me very different code.  It almost seems like a bug because when I step into the code with the debugger, it shows me the source from the symbols but its not actually running that code. Instead, it behaves like its running the compiled binary. I'm seeing weird things when debugging like a function returning when stepping over a line that doesn't even have a return (not from an exception thrown). Has anyone else ever encountered this?

I find this very odd because when I ctrl+click something, it will take me to the decompiled code but when I step into the code from the debugger (which uses the symbols), it gives me very different code

It is possible you're seeing the sourcelink source in the debugger - which is the real, not decompiled source.  We are tracking enabling source link for goto definition here - https://github.com/dotnet/vscode-csharp/issues/6784

Instead, it behaves like its running the compiled binary. I'm seeing weird things when debugging like a function returning when stepping over a line that doesn't even have a return (not from an exception thrown). Has anyone else ever encountered this?

If you're debugging a binary built with optimizations enabled (e.g. release mode) this can happen with either decompiled source or source link source.


Add missing block.nonce == beaconNonce == 0 in consensus validation

Describe the feature
the default block nonce in the beacon consensus is 0, which is currently not enforced in
https://github.com/paradigmxyz/reth/blob/eb37bc7b37b853d9bd11b98ea78652e575fd9466/crates/consensus/common/src/validation.rs#L13-L17
TODO
add check for block nonce
ref https://github.com/ethereum/go-ethereum/blob/8f373227ac481685148019b21ef9e1478d3ba609/consensus/beacon/consensus.go#L232-L235
Additional context
No response

The check is here: https://github.com/paradigmxyz/reth/blob/ec7a10b57e3cf97ce6d12414e2f2266338d61351/crates/consensus/beacon/src/beacon_consensus.rs#L40-L80
The check is related to TD and because of that it is separate.

Will close this, as check is present.

ah, overlooked that!


change repo structure for alerts, collectors

When this is merged, we should change the branch variable in the populate.js script in odyslam/marketplace

@OdysLam any feeling on if this will get done or need more work?
I was hoping to add the aggregator colloector as an example:
https://github.com/andrewm4894/netdata/tree/aggregator-collector/collectors/python.d.plugin/aggregator
Have actually had one or two people seperatly ask for this over last few months so was thinking it could be useful for some potentially. https://community.netdata.cloud/t/any-way-to-aggregate-multple-hosts-without-using-cloud/1669


Add html content check condition

As mentioned in my previous issue #61 i'm currently using healthchecks, and the last feature i would need is some generic content regex / string presence check. now, i might be missing something everybody already knows, since you have this BODY placeholder, and maybe that's just it.
anyway, it seems that the process basically would mean gatus curls the site content and runs a regex or literal string against it, and when it finds it, test passes.
what do you think?
in terms of performance, i think that this is actually the most heavy that healthchecks is doing, but already now i'm running all the tests in parallel, and gatus outperforms healthchecks by magnitudes. so nice.

This is technically already possible by leveraging the placeholder [BODY] with the function pat (short for pattern).
I did just find a small issue with pat due to backslashes, which I'll fix in a bit, but let me know if this is sufficient.
Here's an example:
services:
  - name: example
    url: "https://twinnation.org/"
    interval: 5m
    conditions:
      - "[BODY] == pat(*DOCTYPE*)"

This is technically already possible by leveraging the placeholder [BODY] with the function pat (short for pattern).
I did just find a small issue with pat due to backslashes, which I'll fix in a bit, but let me know if this is sufficient.
Here's an example:
services:
  - name: example
    url: "https://twinnation.org/"
    interval: 5m
    conditions:
      - "[BODY] == pat(*DOCTYPE*)"

ok, interesting. i've put your example in with one of my sites, literally checking for that string DOCTYPE, but for some reason it returns false. weirdly enough on the status page i get a green checkmark, but the check details look like this:
Timestamp:
2021-01-06 16:09:59
Response time:
42ms
Conditions:
X ~ [BODY] == pat(*DOCTYPE*) ( == pat(*DOCTYPE*))

same with the literal string:
Timestamp:
2021-01-06 16:14:02
Response time:
43ms
Conditions:
X ~ [BODY] == pat(DOCTYPE) ( == pat(DOCTYPE))

could i run this test somehow from the command line, as a go oneliner, just to see some more details, how the BODY content looks like etc?

ok, interesting. i've put your example in with one of my sites, literally checking for that string DOCTYPE, but for some reason it returns false. weirdly enough on the status page i get a green checkmark, but the check details look like this:
Timestamp:
2021-01-06 16:09:59
Response time:
42ms
Conditions:
X ~ [BODY] == pat(*DOCTYPE*) ( == pat(*DOCTYPE*))

same with the literal string:
Timestamp:
2021-01-06 16:14:02
Response time:
43ms
Conditions:
X ~ [BODY] == pat(DOCTYPE) ( == pat(DOCTYPE))

could i run this test somehow from the command line, as a go oneliner, just to see some more details, how the BODY content looks like etc?

sidenote: i meant to say i use cabot, not healthchecks for this kind of work previously.

sidenote: i meant to say i use cabot, not healthchecks for this kind of work previously.

@whysthatso Have you run this with the latest version (v1.11.0)?
I fixed an issue caused with pattern caused by backslashes.
Originally, the pat function wasn't intended for HTML, so I hadn't tested with HTML content, but after doing so, I found out that issue and resolved it.

@whysthatso Have you run this with the latest version (v1.11.0)?
I fixed an issue caused with pattern caused by backslashes.
Originally, the pat function wasn't intended for HTML, so I hadn't tested with HTML content, but after doing so, I found out that issue and resolved it.

i'm running this now on the latest 1.11.2, situation unchanged.

i'm running this now on the latest 1.11.2, situation unchanged.

again, is there any way to run gatus in a debug mode to see the content of BODY?

again, is there any way to run gatus in a debug mode to see the content of BODY?

@whysthatso Technically, if a condition does not succeed, it should automatically print the content of the body.
Here's an example:
services:
  - name: example
    url: "http://example.org/"
    interval: 5m
    conditions:
      - "[BODY] == pat(wow*)"

Logs:
2021/01/15 09:27:32 [watchdog][monitor] Monitored serviceName=example; success=false; errors=0; requestDuration=53ms; responseBody=<!doctype html>

    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />

Example Domain

This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.

More information...

@whysthatso Technically, if a condition does not succeed, it should automatically print the content of the body.
Here's an example:
services:
  - name: example
    url: "http://example.org/"
    interval: 5m
    conditions:
      - "[BODY] == pat(wow*)"

Logs:
2021/01/15 09:27:32 [watchdog][monitor] Monitored serviceName=example; success=false; errors=0; requestDuration=53ms; responseBody=<!doctype html>

    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />

Example Domain

This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.

More information...

yes, i've seen something like that briefly before, however at the moment i get only this:
[Condition][evaluate] Condition '[BODY] == pat(*DOCTYPE*)' did not succeed because ' == pat(*DOCTYPE*)' is false

yes, i've seen something like that briefly before, however at the moment i get only this:
[Condition][evaluate] Condition '[BODY] == pat(*DOCTYPE*)' did not succeed because ' == pat(*DOCTYPE*)' is false

okay, my bad. i've pointed the test to a wrong domain... nevermind, i see the fail output now.
however,
X ~ [BODY] == pat() (   .... == pat())

okay, my bad. i've pointed the test to a wrong domain... nevermind, i see the fail output now.
however,
X ~ [BODY] == pat() (   .... == pat())

i've tried a really universal term now
"[BODY] == pat(*div*)"

still all fails...

i've tried a really universal term now
"[BODY] == pat(*div*)"

still all fails...

@whysthatso Is there any chance you could provide me with the body of the page? If there's any confidential information, feel free to exclude it.
If I can reproduce it, I'll fix it, but with the websites I've tried, I haven't run into this issue.
Once I have the body, I'll create a test for it and trim down the body to as small as possible until I stop getting the issue; that'll allow me to pinpoint what exactly the issue is.
i.e.
func TestCondition_evaluateWithBodyHTMLPattern(t *testing.T) {
	var html = `<!DOCTYPE html> <html lang="et"> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />....`
	condition := Condition("[BODY] == pat(*head*)")
	result := &Result{Body: []byte(html)}
	condition.evaluate(result)
	if !result.ConditionResults[0].Success {
		t.Errorf("Condition '%s' should have been a success", condition)
	}
	expectedConditionDisplayed := "[BODY] == pat(*head*)"
	if result.ConditionResults[0].Condition != expectedConditionDisplayed {
		t.Errorf("Condition '%s' should have resolved to '%s', got '%s'", condition, expectedConditionDisplayed, result.ConditionResults[0].Condition)
	}
}

@whysthatso Is there any chance you could provide me with the body of the page? If there's any confidential information, feel free to exclude it.
If I can reproduce it, I'll fix it, but with the websites I've tried, I haven't run into this issue.
Once I have the body, I'll create a test for it and trim down the body to as small as possible until I stop getting the issue; that'll allow me to pinpoint what exactly the issue is.
i.e.
func TestCondition_evaluateWithBodyHTMLPattern(t *testing.T) {
	var html = `<!DOCTYPE html> <html lang="et"> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />....`
	condition := Condition("[BODY] == pat(*head*)")
	result := &Result{Body: []byte(html)}
	condition.evaluate(result)
	if !result.ConditionResults[0].Success {
		t.Errorf("Condition '%s' should have been a success", condition)
	}
	expectedConditionDisplayed := "[BODY] == pat(*head*)"
	if result.ConditionResults[0].Condition != expectedConditionDisplayed {
		t.Errorf("Condition '%s' should have resolved to '%s', got '%s'", condition, expectedConditionDisplayed, result.ConditionResults[0].Condition)
	}
}

@whysthatso Actually, I think I may have found the issue.
Could re-pull the latest docker tag and try again?

@whysthatso Actually, I think I may have found the issue.
Could re-pull the latest docker tag and try again?

did that, but to no avail. i've actually pruned everything, set up from scratch and only ran your test example, with debug on, this is the config:

debug: true
metrics: true
alerting:
  mattermost:
    webhook-url: "https://url.to.mattermost/hooks/<secret>"
    insecure: false

services:
  - name: example
    url: "https://twinnation.org/"
    interval: 30s
    alerts:
      - type: mattermost
        enabled: true
        description: "healthcheck failed 3 times in a row"
        send-on-resolved: true
    conditions:
      - "[BODY] == pat(*DOCTYPE*)"

this is my compose file:
version: "3"
services:
  gatus:
    image: twinproduction/gatus:latest
    container_name: gatus
    ports:
      - "39425:8080"
    volumes:
      - ./config.yaml:/config/config.yaml

this is the output of docker-compose up
root@bbox004:/opt/gatus# docker-compose up
Pulling gatus (twinproduction/gatus:latest)...
latest: Pulling from twinproduction/gatus
bad8fffb4eda: Pull complete
555aa233bb34: Pull complete
a70b215ed0fb: Pull complete
d2fffcad6470: Pull complete
Digest: sha256:c7ebb4332495c72ad3b7fbe29708203072f1e32d939a6745ace7231f53c48381
Status: Downloaded newer image for twinproduction/gatus:latest
Creating gatus ...
Creating gatus ... done
Attaching to gatus
gatus    | 2021/01/16 15:56:03 [config][Load] Reading configuration from configFile=config/config.yaml
gatus    | 2021/01/16 15:56:03 [config][validateAlertingConfig] configuredProviders=[mattermost]; ignoredProviders=[slack messagebird twilio pagerduty custom]
gatus    | 2021/01/16 15:56:03 [config][validateServicesConfig] Validating service 'example'
gatus    | 2021/01/16 15:56:03 [config][validateServicesConfig] Validated 1 services
gatus    | 2021/01/16 15:56:03 [watchdog][monitor] Monitoring serviceName=example
gatus    | 2021/01/16 15:56:03 [controller][Handle] Listening on <IP_ADDRESS>:8080/
gatus    | 2021/01/16 15:56:04 [watchdog][monitor] Monitored serviceName=example; success=false; errors=0; requestDuration=527ms; responseBody=<!DOCTYPE html>
gatus    |
gatus    |
gatus    |     <meta charset="UTF-8" />
gatus    |     <title>TwiNNatioN</title>
gatus    |
gatus    |     <meta property="og:type" content="website" />
gatus    |     <meta property="og:site_name" content="TwiNNatioN" />
gatus    |     <meta name="twitter:card" content="summary" />
gatus    |     <meta name="twitter:title" property="og:title" content="TwiNNatioN" />
gatus    |
gatus    |
gatus    |     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
gatus    |     <meta name="viewport" content="width=device-width, initial-scale=1" />
gatus    |     <meta name="theme-color" content="#DD0000" />
gatus    |     <link rel="manifest" href="/manifest.json" />
gatus    |     <link rel="shortcut icon" href="/favicon.ico" />
gatus    |     <link rel="stylesheet" href="/css/bootstrap.min.css" />
gatus    |     <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.0.9/css/all.css" />
gatus    |     <link href="https://fonts.googleapis.com/css?family=Maven+Pro" rel="stylesheet" />
gatus    |
gatus    |
gatus    |
gatus    |
gatus    |
You need to enable JavaScript to run this app.

gatus    |
gatus    |

gatus    |
gatus    |

gatus    |
TwiNNatioN

gatus    |
gatus    |

gatus    |
gatus    |

gatus    |
gatus    |
gatus    |
gatus    |
gatus    |
gatus    | 2021/01/16 15:56:04 [watchdog][monitor] Waiting for interval=1m0s before monitoring serviceName=example again

there's also an issue with notifying mattermost, but i will open another issue for that if need be:
gatus    | 2021/01/16 16:02:23 [watchdog][handleAlertsToTrigger] Sending mattermost alert because alert with description='healthcheck failed 3 times in a row' has been TRIGGERED
gatus    | 2021/01/16 16:02:23 [watchdog][handleAlertsToTrigger] Ran into error sending an alert: call to provider alert returned status code 400: {"id":"model.incoming_hook.parse_data.app_error","message":"Unable to parse incoming data.","detailed_error":"","request_id":"znoajjaje7gsuphp1yry9tkbcw","status_code":400}

did that, but to no avail. i've actually pruned everything, set up from scratch and only ran your test example, with debug on, this is the config:

debug: true
metrics: true
alerting:
  mattermost:
    webhook-url: "https://url.to.mattermost/hooks/<secret>"
    insecure: false

services:
  - name: example
    url: "https://twinnation.org/"
    interval: 30s
    alerts:
      - type: mattermost
        enabled: true
        description: "healthcheck failed 3 times in a row"
        send-on-resolved: true
    conditions:
      - "[BODY] == pat(*DOCTYPE*)"

this is my compose file:
version: "3"
services:
  gatus:
    image: twinproduction/gatus:latest
    container_name: gatus
    ports:
      - "39425:8080"
    volumes:
      - ./config.yaml:/config/config.yaml

this is the output of docker-compose up
root@bbox004:/opt/gatus# docker-compose up
Pulling gatus (twinproduction/gatus:latest)...
latest: Pulling from twinproduction/gatus
bad8fffb4eda: Pull complete
555aa233bb34: Pull complete
a70b215ed0fb: Pull complete
d2fffcad6470: Pull complete
Digest: sha256:c7ebb4332495c72ad3b7fbe29708203072f1e32d939a6745ace7231f53c48381
Status: Downloaded newer image for twinproduction/gatus:latest
Creating gatus ...
Creating gatus ... done
Attaching to gatus
gatus    | 2021/01/16 15:56:03 [config][Load] Reading configuration from configFile=config/config.yaml
gatus    | 2021/01/16 15:56:03 [config][validateAlertingConfig] configuredProviders=[mattermost]; ignoredProviders=[slack messagebird twilio pagerduty custom]
gatus    | 2021/01/16 15:56:03 [config][validateServicesConfig] Validating service 'example'
gatus    | 2021/01/16 15:56:03 [config][validateServicesConfig] Validated 1 services
gatus    | 2021/01/16 15:56:03 [watchdog][monitor] Monitoring serviceName=example
gatus    | 2021/01/16 15:56:03 [controller][Handle] Listening on <IP_ADDRESS>:8080/
gatus    | 2021/01/16 15:56:04 [watchdog][monitor] Monitored serviceName=example; success=false; errors=0; requestDuration=527ms; responseBody=<!DOCTYPE html>
gatus    |
gatus    |
gatus    |     <meta charset="UTF-8" />
gatus    |     <title>TwiNNatioN</title>
gatus    |
gatus    |     <meta property="og:type" content="website" />
gatus    |     <meta property="og:site_name" content="TwiNNatioN" />
gatus    |     <meta name="twitter:card" content="summary" />
gatus    |     <meta name="twitter:title" property="og:title" content="TwiNNatioN" />
gatus    |
gatus    |
gatus    |     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
gatus    |     <meta name="viewport" content="width=device-width, initial-scale=1" />
gatus    |     <meta name="theme-color" content="#DD0000" />
gatus    |     <link rel="manifest" href="/manifest.json" />
gatus    |     <link rel="shortcut icon" href="/favicon.ico" />
gatus    |     <link rel="stylesheet" href="/css/bootstrap.min.css" />
gatus    |     <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.0.9/css/all.css" />
gatus    |     <link href="https://fonts.googleapis.com/css?family=Maven+Pro" rel="stylesheet" />
gatus    |
gatus    |
gatus    |
gatus    |
gatus    |
You need to enable JavaScript to run this app.

gatus    |
gatus    |

gatus    |
gatus    |

gatus    |
TwiNNatioN

gatus    |
gatus    |

gatus    |
gatus    |

gatus    |
gatus    |
gatus    |
gatus    |
gatus    |
gatus    | 2021/01/16 15:56:04 [watchdog][monitor] Waiting for interval=1m0s before monitoring serviceName=example again

there's also an issue with notifying mattermost, but i will open another issue for that if need be:
gatus    | 2021/01/16 16:02:23 [watchdog][handleAlertsToTrigger] Sending mattermost alert because alert with description='healthcheck failed 3 times in a row' has been TRIGGERED
gatus    | 2021/01/16 16:02:23 [watchdog][handleAlertsToTrigger] Ran into error sending an alert: call to provider alert returned status code 400: {"id":"model.incoming_hook.parse_data.app_error","message":"Unable to parse incoming data.","detailed_error":"","request_id":"znoajjaje7gsuphp1yry9tkbcw","status_code":400}

@whysthatso I was able to reproduce the issue by pulling twinproduction/gatus:latest, and it turns out that yesterday's Docker build job got stuck, which is why the issue was still present.
I've retried the Docker build task, and the latest tag has now been replaced by the one with fix I mentioned yesterday.
I apologize for the inconvenience, it should work this time (excluding the mattermost issue, which I'll look at separately).

@whysthatso I was able to reproduce the issue by pulling twinproduction/gatus:latest, and it turns out that yesterday's Docker build job got stuck, which is why the issue was still present.
I've retried the Docker build task, and the latest tag has now been replaced by the one with fix I mentioned yesterday.
I apologize for the inconvenience, it should work this time (excluding the mattermost issue, which I'll look at separately).

yey, you did it. it works now nice.

yey, you did it. it works now nice.

Glad to hear that.
Since pat fulfills your use case, would you be fine with me closing this issue?

Glad to hear that.
Since pat fulfills your use case, would you be fine with me closing this issue?

The fix for the pattern function has been released in v1.12.0

The fix for the pattern function has been released in v1.12.0


nostr: URIs doesn't work in bio

Was editing my bio - https://stacker.news/kristapsk.
If I change my Nostr description to "nostr:npub1vxpwwydpmmc4usmxf2zkr8shdjq888g532ydyls3nadxxd9wyx5ssjq5nd" (like a proper nostr URI), it becomes hidden and not visible / clickable.

Nice catch! Thanks for reporting

Hi, apparently the remarkDirective plugins was causing this problem. I've opened a draft PR which can be tested locally for confirmation.

Can this issue be closed? It looks like the net changes from #508 have landed in master.

Yep, let's close