HAP R17 and Homekit Certification Assistant 5.0

Hi there,
Over the next few weeks I'll be working on a fork of this library to integrate a GarageDoorOpener Accessory (via a Bridge).
Running an instance of the latest code (version 2.0.6) against Homekit Certification Assistant 5 results in a few HAP R17 conformance errors - including (among others):

HomekitBridge must have a Protocol Information Service
GarageDoorOpener characteristic format types should be uint8 (currently int)
Missing support for error responses via HTTP 207 MultiStatusResponse

Are there any plans for these in the next development iteration (2.0.7?). And if not, would you like me to create Pull Requests for fixes that I implement locally?

There are not currently any plans to implement any of those features, and PRs would be welcome.
I do have some questions for you though that possibly shouldn't be discussed publicly. Please email me at<EMAIL_ADDRESS>


Update to opencollective-postinstall

The https://github.com/opencollective/opencollective-cli package is
deprecated, and
https://github.com/opencollective/opencollective-postinstall is a more
lightweight alternative, with no transitive dependencies.

Another option now that requires no packages is to use the 'funding' field in package.json:
https://blog.opencollective.com/beyond-post-install/
I can update to do that instead, if you prefer.


Split SE by colData group

There is a need to split SE objects by sample groups, typically defined by a colData field (subjectID, for instance).
Searches do not return anything readily available. We could implement the following in mia or make a PR in SummarizedExperiment. The latter may take a longer time?
library(mia)
data(GlobalPatterns)
se <- GlobalPatterns
field <- "SampleType"

# ideally, grouping could be either vector/factor of groups, or a name of colData field
splitSE <- function(se, grouping, use.names=TRUE) {
  # Sample indices for each group
  spl <- split(1:ncol(se), colData(se)[[grouping]])
  se_list <- lapply(seq_along(spl), function(i) se[,spl[[i]]])
  if (use.names) {names(se_list) <- names(spl)}
  return (se_list)
}
se_list <- splitSE(se, field)

This works little bit similarly to splitByRanks. (altExp cannot be used in this case, but maybe MultiAssayExperiment?) We could add this to mia, at least first. --> splitByGroups?

MAE is not optimal either because to some extent it assumes that there can be matching between samples, even if it is not one-to-one. If there is a good data structure available that supports this then that would be optimal. If we do not identify such structure, then a simple list of SE objects might be the easiest solution. We can give a try to MAE, and if it seems to work well in practice, anyway, then whynot. Conceptually it might be a bit confusing, however, since in a typical use case the samples have some matching, and the different experiments correspond to different views or measurements for the same/matched samples.

splitBy would be a good name.
The function could use a defined factor or a character name for colData to get the factor. In addition the length of the factor could determine if cols or rows are split, either by the length or whether the name is found in col/rowData. colData should take precedence, since the split function is already usable for rows.

When SE is split by rowData (e.g. taxonomy), one can use altExps because samples remain matched. This is how splitByRanks operates, for instance.
When SE is split by colData (e.g. whatever grouping there), one cannot use altExps because samples are, in general, not matched after the split by an arbitrary grouping factor.
Then would the output from splitBy be a list of SE objects? This would be different than the output from splitByRanks.

When SE is split by rowData (e.g. taxonomy), one can use altExps because samples remain matched. This is how splitByRanks operates, for instance.
When SE is split by colData (e.g. whatever grouping there), one cannot use altExps because samples are, in general, not matched after the split by an arbitrary grouping factor.
Then would the output from splitBy be a list of SE objects? This would be different than the output from splitByRanks.

Sure. It is a differenc operation, therefore a different output is expected, isn't it?
Btw: splitByRanks also returns a list of SEs so It guess it is not that different after all

Sure. It is a differenc operation, therefore a different output is expected, isn't it?
Btw: splitByRanks also returns a list of SEs so It guess it is not that different after all

splitBy with rowData is different from splitByRanks but it could be put into altExp, unlike splitBy with colData.
It might be conceptually more clear that splitBy returns a list in both cases. The utilities of altExp are not generally applicable in this case but that's probably fine for this one.

splitBy with rowData is different from splitByRanks but it could be put into altExp, unlike splitBy with colData.
It might be conceptually more clear that splitBy returns a list in both cases. The utilities of altExp are not generally applicable in this case but that's probably fine for this one.


Cone angular diff

There seems to be an issue with the log_ang_diff variable for the diffuse gamma file, see plot below. I would have expected a similar distribution to the on-source file. Or is this expected?

This issue turns out to be related to how the angles are calculated on the sky. We probably have to apply some correction to get to Ze/Az and then calculate the angular difference.
However, since this is not really important for our case, we simply changed to calculating the angular difference on the camera. That avoids this issue.
So, since this issue is no longer relevant for our study, I will close it.


Cannot Load Texture Onto Body in Gatsby React Component

This is a continuation from a theme first identified in 2017:
https://github.com/liabru/matter-js/issues/497#issuecomment-519113260
I'm having trouble loading a sprite into a Matter JS enabled component inside of React within a Gatsby project.
The component loads fine when I set options: { wireframes: true }. Otherwise it just freezes when I set: { wireframes: false }
// create a renderer
        var render = Render.create({
            element: this.refs.avalanche,
            engine: engine,
            options: {
                width: 600,
                height: 600,
                showAngleIndicator: true,
                wireframes: false
            }
        });

         // add bodies
         var stack = Composites.stack(20, 20, 20, 5, 0, 0, function(x, y) {
            return Bodies.circle(x, y, Common.random(10, 20), {
                friction: 0.00001,
                restitution: 0.5,
                density: 0.001,
                render: {
                    sprite: {
                        texture: './spring.png'
                    }
                }
            });
        });

I have confirmed that the path to the PNG is working.
The example code below is a hybrid taken from the Avalanche and Sprites example projects on the Matter.js demo set.
The full component code is below.
import React, { Component } from 'react';
import Matter from 'matter-js';

class Avalanche extends Component {
    constructor(props) {
        super(props);
        this.state = {};
    }

    componentDidMount() {
        // module aliases
        var Engine = Matter.Engine,
            Render = Matter.Render,
            Runner = Matter.Runner,
            Composite = Matter.Composite,
            Composites = Matter.Composites,
            Common = Matter.Common,
            MouseConstraint = Matter.MouseConstraint,
            Mouse = Matter.Mouse,
            World = Matter.World,
            Bodies = Matter.Bodies;

        // create an engine
        var engine = Engine.create(),
            world = engine.world;

        // create a renderer
        var render = Render.create({
            element: this.refs.avalanche,
            engine: engine,
            options: {
                width: 600,
                height: 600,
                showAngleIndicator: true,
                wireframes: false
            }
        });

        // run the renderer
        Render.run(render);

         // create runner
        var runner = Runner.create();
        Runner.run(runner, engine);

         // add bodies
         var stack = Composites.stack(20, 20, 20, 5, 0, 0, function(x, y) {
            return Bodies.circle(x, y, Common.random(10, 20), {
                friction: 0.00001,
                restitution: 0.5,
                density: 0.001,
                render: {
                    sprite: {
                        texture: './sprite.png'
                    }
                }
            });
        });

        World.add(world, stack);

        World.add(world, [
            Bodies.rectangle(200, 150, 700, 20, { isStatic: true, angle: Math.PI * 0.06 }),
            Bodies.rectangle(500, 350, 700, 20, { isStatic: true, angle: -Math.PI * 0.06 }),
            Bodies.rectangle(340, 580, 700, 20, { isStatic: true, angle: Math.PI * 0.04 })
        ]);

        var mouse = Mouse.create(render.canvas),
            mouseConstraint = MouseConstraint.create(engine, {
            mouse: mouse,
            constraint: {
                stiffness: 0.2,
                render: {
                    visible: false
                }
            }
        });

        World.add(world, mouseConstraint);

        // keep the mouse in sync with rendering
        render.mouse = mouse;

        // fit the render viewport to the scene
        Render.lookAt(render, Composite.allBodies(world));

        // wrapping using matter-wrap plugin
        for (var i = 0; i < stack.bodies.length; i += 1) {
            stack.bodies[i].plugin.wrap = {
                min: { x: render.bounds.min.x, y: render.bounds.min.y },
                max: { x: render.bounds.max.x, y: render.bounds.max.y }
            };
        }

    }

    render() {
        return <div ref="avalanche" />;
    }
};

export default Avalanche;

I've discovered the cause of the bug.
The Composites.js library will only work if Render.options.wireframes=true.
There is no meaningful error message coming back from the library, but I've pasted the part of the code that leads to the issue.
    var render = Render.create({
      element: this.refs.scene,
      engine: engine,
      options: {
        width: 600,
        height: 600,
        wireframes: false
      }
    });

// the following code will break the app
    var stack = Composites.stack(20, 20, 20, 5, 0, 0, function(x, y) {
        return Bodies.circle(x, y, Common.random(10, 20), {
            friction: 0.00001,
            restitution: 0.5,
            density: 0.001,
            render: {
                sprite: {
                    texture: 'btcIcon.png',
                    xScale: 100,
                    yScale: 100
                }
            }
        });
    });

My current workaround is to create a lot of manual bodies with sprites and then just import them into the engine.
var ballA = Bodies.circle(210, 100, 30, {
    restitution: 0.5,
    friction: 0.00001,
    density: 0.001,
    render: {
        sprite: {
            xScale: 0.1,
            yScale: 0.1,
            texture: 'btcIcon.png'
        }
    }
});
    var ballB = Bodies.circle(110, 50, 30, {
        restitution: 0.5,
        friction: 0.00001,
        density: 0.001,
        render: {
            sprite: {
                xScale: 0.1,
                yScale: 0.1,
                texture: 'btcIcon.png'
            }
        }
    });
    World.add(engine.world, [ballA, ballB]);

Closing as it looks like you managed to solve this one, thanks for sharing.


as a project lead I would like to be able to have the clusternator initialize an AWS network without doing a full clusternator init

Expose a minimal clusternator init that just checks the subnet, route tables, and ACL.

The use case for this is:
Clusternator adds new AWS feature, like say fancy logging, but legacy clusternator projects don't have the relevant AWS features setup, and need a quick way of setting them up, without obliterating their current clusternator settings.


Added another error to the table

several customers have asked to include the listed error in this Doc. the latest example is SR#<PHONE_NUMBER>02238

@willbrim : Thanks for your contribution! The author, @MicrosoftGuyJFlo, has been notified to review your proposed change.

#sign-off


Generate manifest

Tool should be able to generate a manifest for Celix based on given input/configuration/defaults.

Fixed in f0f14202c482214ad88df00fc2a6059ffab9d558, closing.


Ability to set the proxy URL and the metadata object

Hi,
I love this library!
I want to ask for some improvements:

add the ability to set the proxy URL
I want to cache the preview results (the metadata JSON), so it can be nice to pass the fetched object itself instead of the fetcher.
Add onFailure callback which being called when failed to get the metadata

Thanks

Hey, please feel free to put in a PR for these features if you want and I will merge it :)


Add Netlify webhook field in WordPress ACF options page

See here: https://github.com/distilagency/koval/issues/7

Added in Netlify Connect. https://github.com/distilagency/netlify-connect/blob/master/includes/class-lbn-settings.php#L94


Update ios sauce comments

This repo is no longer in use. Check out the sample-code in the main appium repo now instead.


Free Logo Maker

tailorbrands.com

From their FAQs, only if you purchase the design provided by the logo maker you can use it!


Ability to inspect specific file?

Would be nice being able to configure a specific file to be inspected by dependabot.
In my repository there's a setup.py and a requirements-dev folder (with additional dev dependencies only used locally). Even though my configuration looks like
version: 1
update_configs:
  - package_manager: "python"
    directory: "/"
    update_schedule: "daily"

the bot is also inspecting files in requirements-dev, which I would expect to be inspected if my configuration looked rather like:
update_configs:
  - package_manager: "python"
    directory: "/requirements-dev"
    update_schedule: "daily"

At the same time, as I'm using pip-tools, I'd like dependabot to inspect only the .ini file as the .txt is automatically generated

Is this possible? It would make configuration much easier

Is this possible? It would make configuration much easier

Indeed, never got any reply on this


Overhaul paginated content navigation

Please describe the problem that your request should resolve.
Currently paginated always loads 3 records on mobile and 10 records in the desktop. I propose the following changes:

Make the number of records displayed on the desktop to be a multiple of mobile numbers, so we can always fetch the same number of records for both resolutions while hiding some of them with CSS if necessary. That will allow us to keep the whole rendering on the SSR, regardless of the resolution.
Make it possible to change the number of displayed records e.g. e.g. like in category pages in e-commerce
Since some of the records may change pretty often - add a refresh button without reloading the whole page.
Save the page that user sees in the URL so navigating to other pages could be in line with browser history

Investigate what is the page index for because it works also without it

related https://github.com/aeternity/aescan/issues/786


Fine-Tune ColBERT in Google Colab-Notebook

Hello,
I am trying to fine-tune the ColBERTv2 pre-trained checkpoint with additional data in a Google Colab-Notebook using the following code:
with Run().context(RunConfig(nranks=1)):
  config = ColBERTConfig(bsize=32, lr=3e-06, warmup=None, doc_maxlen=180, dim=128, nway=2, accumsteps=1, use_ib_negatives=False)
  trainer = Trainer(
    triples='triples.tsv',
    queries='queries.tsv',
    collection='documents_train.tsv',
    config=config,
  )

  checkpoint_path = trainer.train(checkpoint=path_to_colbertv2.0)

The code is stuck on #> Starting.... What can I do to make the training progress?

We may need more info on the error to be able to help. The error seems to be masked in the child process because it's running in colab. Basically training forks a new process and the error isn't getting propagated back to the parent, as far as I can tell from this

Thank you for your answer!
Do you know if there is any way to get more info on the error in Colab?

I figured it out: My triples-file was not in the correct JSONL-format which lead to a trailing data error.

Hey @franziskaweber  how did you know you had an error? For me i do not see any logs other than an eternal #> Starting...

@franziskaweber did you face a similar issue as this?
https://github.com/stanford-futuredata/ColBERT/issues/254#issue-1904577165 #254

Hey @roynirmal,
I also only saw the eternal #> Starting.... But once I fixed the format of my triples-file, everything worked.

What did you change the format too?  I tried everything (including making it exactly as similar as the ms-marco files)

The file documents_train.tsv contains the training documents and has the following format:
0 text0
1 text1
2 text2
...
The file queries.tsv contains the queries and has the following format:
0 query0
1 query1
2 query2
...
The file triples.json contains the training triples and has the following format:
[0, 12692, 48199]
[0, 12693, 17664]
[1, 23783, 21169]
[1, 3796, 17104]
[1, 47897, 30084]
[2, 52241, 54085]
...

The first component is the id of the query, the second one the id of a relevant document and the third one the id of an irrelevant document.
I hope this helps!

The file documents_train.tsv contains the training documents and has the following format:
0 text0
1 text1
2 text2
...

The file queries.tsv contains the queries and has the following format:
0 query0
1 query1
2 query2
...

The file triples.json contains the training triples and has the following format:
[0, 12692, 48199]
[0, 12693, 17664]
[1, 23783, 21169]
[1, 3796, 17104]
[1, 47897, 30084]
[2, 52241, 54085]
...
The first component is the id of the query, the second one the id of a relevant document and the third one the id of an irrelevant document.

I hope this helps!

I have the exact same issue and have even dummied it down to documents with only 10 lines. I am using the example in the readme for training and have collection/query TSV as mentioned above.
But when i try to train in colab it stalls on starting. I have tried having the triplets constructed as mentioned above in both TSV and JSONL format.
In the comment you wrote .json, does extension have any impact and is curly braces needed somewhere in the file?

I didn't use any curly braces in the triples-file, just square braces as described above. You can try changing your extension to .json but I don't think that it will have any impact.
I used to following command to create the JSON-file:
triples_file.write(json.dumps([query_id, rel_doc_id, irrel_doc_id])  + '\n')
You can try using exactly the same command.
Otherwise, I really can't say what goes wrong in your case. Maybe the problem isn't even the triples-file but something else.


Compatibility with Redis 7

Is the library ready to function correctly with Redis 7?

Yes, rueidis works with Redis 7


mobile peers download only

Hello,
how to configure the p2pml config that some peers (for example mobile) only download (p2p / http) ?
its possible ?
Thanks.

Sure, but p2p-media-loader itself has single responsibility - download media via P2P. Everything else you can easily implement on top of it:
var player;
if (isMobile()) { // Create Clappr with P2P
    var engine = new p2pml.hlsjs.Engine();

    player = new Clappr.Player({
        parentId: "#player",
        source: "https://akamai-axtest.akamaized.net/routes/lapd-v1-acceptance/www_c4/Manifest.m3u8",
        mute: true,
        autoPlay: true,
        playback: {
            hlsjsConfig: {
                liveSyncDurationCount: 7,
                loader: engine.createLoaderClass()
            }
        }
    });

    p2pml.hlsjs.initClapprPlayer(player);
} else { // Create Clappr without P2P
    player = new Clappr.Player({
        parentId: "#player",
        source: "https://akamai-axtest.akamaized.net/routes/lapd-v1-acceptance/www_c4/Manifest.m3u8",
        mute: true,
        autoPlay: true
    });
}

Ok, its no problem, but
how to configure mobile peers that they only download
( through p2p)
and were not seeders ?
we cannot understand how :-(

Implemented consumeOnly configuration parameter:
const engine = new p2pml.hlsjs.Engine({
    loader: {
        consumeOnly: isMobile()
    }
});


torchsummary or torchstat

When I use torchsummary or torchstat to calculate the parameters, I meet
Traceback (most recent call last):
File "./tools/train.py", line 167, in
main()
File "./tools/train.py", line 137, in main
stat(model, (3, 256, 256))
File "/usr/local/lib/python3.6/dist-packages/torchstat/statistics.py", line 71, in stat
ms.show_report()
File "/usr/local/lib/python3.6/dist-packages/torchstat/statistics.py", line 64, in show_report
collected_nodes = self._analyze_model()
File "/usr/local/lib/python3.6/dist-packages/torchstat/statistics.py", line 57, in _analyze_model
model_hook = ModelHook(self._model, self._input_size)
File "/usr/local/lib/python3.6/dist-packages/torchstat/model_hook.py", line 24, in init
self._model(x)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/gaozishu/data/mmdetection_550/mmdet/core/fp16/decorators.py", line 51, in new_func
return old_func(*args, **kwargs)
TypeError: forward() missing 1 required positional argument: 'img_metas'
What should I do? Thank you

This is because the model forward requires img_metas as input. To get the number of parameters, you can use this script.

This is because the model forward requires img_metas as input. To get the number of parameters, you can use this script.

Hi, the script link is not available now, could you please update the link to the script? Thanks!


Feature Request: save all stderr

Go program will print goroutine stack to stderr under fatal error, which will bypass log files.
Should redirect stderr in run script.
To avoid disk space running out, redirect with > is acceptable.
Should not use >> for append.
Should not append to same log file as main program.

已添加 stderr log: https://github.com/pingcap/tidb-ansible/pull/71


Compile fail on CentOS

I am trying to compile BART 0.5.00 on a CentOS cluster. I cloned the master branch from git, and encountered the following error:
gcc -Wall -Wextra -MMD -MF /var/scratch/tbbakker/bart/src/num/.lapack.d -iquote /var/scratch/tbbakker/bart/src/ -I/usr//include/ -I/usr//include -DFFTWTHREADS -DMAIN_LIST="avg, bench, bitmask, cabs, caldir, calmat, carg, casorati, cc, ccapply, cdf97, circshift, conj, conv, copy, cpyphs, creal, crop, delta, ecalib, ecaltwo, estdelay, estdims, estshift, estvar, extract, fakeksp, fft, fftmod, fftrot, fftshift, filter, flatten, flip, fmac, homodyne, index, invert, itsense, join, looklocker, lrmatrix, mandelbrot, mip, moba, nlinv, noise, normalize, nrmse, nufft, ones, pattern, phantom, pics, pocsense, poisson, poly, repmat, reshape, resize, rmfreq, rof, rss, sake, saxpy, scale, sdot, show, slice, spow, sqpics, squeeze, ssa, std, svd, tgv, threshold, toimg, traj, transpose, twixread, var, vec, version, walsh, wave, wavelet, wavepsf, whiten, window, wshfl, zeros, zexp, ()" -include src/main.h -O3 -ffast-math -Wmissing-prototypes -std=gnu11 -fopenmp -c -o /var/scratch/tbbakker/bart/src/num/lapack.o /var/scratch/tbbakker/bart/src/num/lapack.c /var/scratch/tbbakker/bart/src/num/lapack.c:14:21: fatal error: lapacke.h: No such file or directory  #include <lapacke.h>                      ^ compilation terminated. make[1]: *** [/var/scratch/tbbakker/bart/src/num/lapack.o] Error 1
I don't have root access on this system, so could not follow the instructions detailed in the README for installation on CentOS:
sudo yum install devtoolset-8 atlas-devel fftw3-devel libpng-devel lapack-devel
Instead, given the error, I tried building lapack (version 3.9.0) from source. Unfortunately, the error remains.
I tried inspecting the Makefile for a hint, and found the following lines:
ifeq ($(BUILDTYPE), Linux) ifneq (,$(findstring Red Hat,$(shell gcc --version))) 	CPPFLAGS+=-I/usr/include/lapacke/ 	LDFLAGS+=-L/usr/lib64/atlas -laterals endif endif
I suspect this tells the compiler where to look for the missing lapacke.h? As a side-note, this line was missing from the Makefile in bart-0.5.00.tar I obtained from the BART webpage in another attempt to solve this issue (is this intentional?).
Regardless, /usr/include/lapacke/ indeed exists on this system, and contains (among other files) a file lapacke.h. I'm rather unfamiliar with these systems, and am unsure how to proceed.
CentOS version: CentOS Linux release 7.4.1708 (Core)
BART version: 0.5.00

Try to set NOLAPACKE=1 in the Makefile.local

Thanks for the quick reply!
That definitely did something, I now get the following error:
gcc -rdynamic -O3 -ffast-math -Wmissing-prototypes -std=gnu11 -fopenmp -Wall -Wextra -DNOLAPACKE -MMD -MF ./.bart.d -iquote /var/scratch/t bbakker/bart/src/ -Isrc/lapacke -I/usr//include/ -I/usr//include -DFFTWTHREADS -DMAIN_LIST="avg, bench, bitmask, cabs, caldir, calmat, car g, casorati, cc, ccapply, cdf97, circshift, conj, conv, copy, cpyphs, creal, crop, delta, ecalib, ecaltwo, estdelay, estdims, estshift, es tvar, extract, fakeksp, fft, fftmod, fftrot, fftshift, filter, flatten, flip, fmac, homodyne, index, invert, itsense, join, looklocker, lr matrix, mandelbrot, mip, moba, nlinv, noise, normalize, nrmse, nufft, ones, pattern, phantom, pics, pocsense, poisson, poly, repmat, resha pe, resize, rmfreq, rof, rss, sake, saxpy, scale, sdot, show, slice, spow, sqpics, squeeze, ssa, std, svd, tgv, threshold, toimg, traj, tr anspose, twixread, var, vec, version, walsh, wave, wavelet, wavepsf, whiten, window, wshfl, zeros, zexp, ()" -include src/main.h -Dmain_re al=main_bart -o bart src/main.c /var/scratch/tbbakker/bart/src/bart.o lib/libbox.a lib/libgrecon.a lib/libsense.a lib/libnoir.a lib/libite r.a lib/liblinops.a lib/libwavelet.a lib/liblowrank.a lib/libnoncart.a lib/libcalib.a lib/libsimu.a lib/libsake.a lib/libdfwavelet.a lib/l ibnlops.a lib/libmoba.a lib/libnum.a lib/libmisc.a lib/libnum.a lib/libmisc.a lib/liblapacke.a -L/usr//lib -lfftw3f -lfftw3f_threads  -L/u sr//lib -llapack -lblas -lpng   -lm
/cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libm.so when searching for -lm /cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libm.so when searching for -lm /cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libgcc_s.so.1 when searching for libgcc_s.so.1 /cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libgcc_s.so.1 when searching for libgcc_s.so.1 /cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libpthread.so when searching for -lpthread /cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libpthread.so when searching for -lpthread /cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libc.so when searching for -lc /cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libc.so when searching for -lc /cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libgcc_s.so.1 when searching for libgcc_s.so.1 /cm/shared/package/gcc/4.9.2/bin/ld: skipping incompatible /usr//lib/libgcc_s.so.1 when searching for libgcc_s.so.1
lib/libnum.a(blas.o): In function `blas_cgemm': blas.c:(.text+0x7e): undefined reference to `cblas_cgemm' lib/libnum.a(blas.o): In function `blas_matrix_multiply': blas.c:(.text+0xf7): undefined reference to `cblas_cgemm' lib/libnum.a(blas.o): In function `blas_csyrk': blas.c:(.text+0x18c): undefined reference to `cblas_csyrk' collect2: error: ld returned 1 exit status make[1]: *** [bart] Error 1
Followed by a bunch of Field 'name' not cached: and Field 'hname' not cached:.

There are packages for CentOS 8. Does this help?
https://copr.fedorainfracloud.org/coprs/schaten/bart/

We have new tests and updated README for installing on different Linux machines.. Please reopen if necessary.


0.16.0 Formatting and address linting errors

For #577
Ran formatter and addressed linting errors.
I added _ underscores to the unused arguments but I would reccomend they just be removed and add a comment or TODO if they were meant as placeholders.
Messages
========

lifelines/fitters/coxph_fitter.py
  Line: 619
    pylint: unused-argument / Unused argument 'weights' (col 43)
  Line: 717
    pylint: unused-variable / Unused variable 'weighted_average' (col 12)
  Line: 764
    pylint: unused-argument / Unused argument 'T' (col 46)
  Line: 1327
    pylint: redefined-builtin / Redefining built-in 'help' (col 32)

Check Information
=================
         Started: 2018-12-21 06:46:14.057368
        Finished: 2018-12-21 06:47:00.913090
      Time Taken: 46.86 seconds
       Formatter: grouped
        Profiles: .prospector.yaml, strictness_medium, strictness_high, strictness_veryhigh, no_doc_warnings, no_member_warnings, no_test_warnings
      Strictness: from profile
  Libraries Used:
       Tools Run: dodgy, mccabe, pep8, profile-validator, pyflakes, pylint, pyroma
  Messages Found: 4

After addressing the above issues I'm now seeing this...
I didn't want to suppress this in-case it has something to do with the failing unit-tests.
lifelines/generate_datasets.py
  Line: 26
    pylint: invalid-unary-operand-type / bad operand type for unary -: result (col 20)
    pylint: invalid-unary-operand-type / bad operand type for unary -: tuple (col 20)

@CamDavidsonPilon Just FYI this does nothing to address the failing unit-tests

Ooooh nice @Kilo59 !
One question: why have you changed everywhere from single to double quotes?  Both are fine and Im not aware of any strong recommendation to use either one or the other

This was done by the black python formatter.
https://github.com/ambv/black#strings

The main reason to standardize on a single form of quotes is aesthetics. Having one kind of quotes everywhere reduces reader distraction. It will also enable a future version of Black to merge consecutive string literals that ended up on the same line (see #26 for details).
Why settle on double quotes? They anticipate apostrophes in English text. They match the docstring standard described in PEP 257. An empty string in double quotes ("") is impossible to confuse with a one double-quote regardless of fonts and syntax highlighting used. On top of this, double quotes for strings are consistent with C which Python interacts a lot with.
On certain keyboard layouts like US English, typing single quotes is a bit easier than double quotes. The latter requires use of the Shift key. My recommendation here is to keep using whatever is faster to type and let Black handle the transformation.

As they reccomend I would suggest you keep writing however you are comfortable and then make use of the Makefile (if you are on Linux/Mac) to format things for you.
# install requirements
make
# call the make format command
make format

If you are on Windows you can use black directly.
In this case . just tells black to format everything in the currect directory but you can pass it a specific path as well.
pip install black
black . --line-length 120

Thanks @Kilo59! Part of my workflow is to code "naturally", and then lint as a close-to-final-step (or when I feel bored).

@Kilo59
Thanks for the explanation, learned something new!


problem implementing external serialisation for std::chrono::timepoint

Trying to implement serialization  functions to handle std::chrono::timepoint, but to no avail.  I've already handled a bunch of custom objects, like boost::circular_buffer, llvm::SmallString, so I though I got the hang of it, seems not.
Here is the code.
https://godbolt.org/z/eWTGhq3zz
Any suggestion much appreciated.

The external serialize(..) function needs to be either in the class namespace, or in the namespace zpp::bits, here because the std::chrono::time_point is inside std, I recommend to put the serialize(..) function in the zpp::bits namespace because it would be illegal to add it to the std namespace (by the language rules). But, you also need to add a custom serialize function for std::chrono::duration in the same way to make it work because you have defined only for time_point.

That was it , thank you.  Great piece of software, congratulation.


[core] Replace PathMatcher#findEntries full-copy list result with lazy stream

A continuation of micro optimizations (#1942).
This time, I've changed the way how PathMatcher returns matched entries - we're using this method 3 times in our request lifecycle:

https://github.com/javalin/javalin/blob/48cd9eb25ac19c2838b487c59573e5cbd107f621/javalin/src/main/java/io/javalin/http/servlet/DefaultTasks.kt#L18
https://github.com/javalin/javalin/blob/48cd9eb25ac19c2838b487c59573e5cbd107f621/javalin/src/main/java/io/javalin/http/servlet/DefaultTasks.kt#L24
https://github.com/javalin/javalin/blob/48cd9eb25ac19c2838b487c59573e5cbd107f621/javalin/src/main/java/io/javalin/http/servlet/DefaultTasks.kt#L68

The 3 full-copy lists are replaced with 3 streams, as long as it doesn't matter that much for BEFORE/AFTER (because we're iterating through the whole result anyway), it'll be definitely an upgrade for http scope that requests only the first element (firstOrNull ) and ignores the rest of elements. Stream is used over Sequence, because Sequence cannot be used by Java users.
The results for a single pathMatcher.findEntries().first() call:
Benchmark                               Mode  Cnt         Score   Error  Units
// old:
PathMatcherBenchmark.matchFirstList    thrpt    2    398864,329          ops/s
PathMatcherBenchmark.matchLastList     thrpt    2    384358,609          ops/s
// new:
PathMatcherBenchmark.matchFirstStream  thrpt    2  30094664,631          ops/s
PathMatcherBenchmark.matchLastStream   thrpt    2    414129,483          ops/s

Thank you very much @dzikoysk 🙇♂


Matrix3.IDENTITY fails to transform nodes properly on IE9 and IE10

There are 2 places where setting Matrix3.IDENTITY on a node has buggy behavior (including during resetTransform()), but setting Matrix3.scaling( 1, 1 ) has the correct behavior.  For instance, see https://github.com/phetsims/forces-and-motion-basics/issues/32
This seems like a high priority problem to me.  I'll keep using Matrix3.scaling(1,1) as a workaround until we can find a superior solution.

We no longer support IE9 or IE10, closing


404 on paypalobjects.com/api/checkout.4.0.133.js

Locally, I'm able to successfully run a Capybara & Rspec test suite which goes through the flow of using paypal-checkout with Braintree processing the transaction.  However, on CircleCI, when I run the same suite, I get the following:
https://www.paypalobjects.com/api/checkout.4.0.133.js - Failed to load resource: the server responded with a status of 404
This occurs on every capybara spec which goes through the paypal checkout flow. And happens when trying to find the .paypal-button-content element:
within_frame find('.paypal-button iframe') do
  find('.paypal-button-content').click
end

Since the iframe is found, something is loading from paypal, but between turning the container into an iframe and finding the button, it fails to load checkout.4.0.133.js.
The test suite is being run on a headless version of chrome. Ruby 2.3.5 is running the backend, and Node 6.6.0 is running webpack 2.1.0-beta.27 to build the frontend.
Both locally and on CircleCI, I'm using the Braintree & PayPal sandbox.
I'm using npm packages braintree-web version 3.22.2, and paypal-checkout version 4.0.125.
Happy to provide more details, but I'm not quite sure which details would be helpful since so much of the login is inside those two packages.

Hi @tzusman, thanks for the heads up. Looks like there was a deploy issue with https://www.paypalobjects.com/api/checkout.4.0.133.js, I kicked off another and the file should be up now.


Is this repository maintained?

I'm upgrading to React 16 in my development machine and it's seems like it's reactable is incompatible with React 16. Considering the last commit on this repo i'm worried it won't be maintained to support React 16. I wanna discuss this here so that if it's not going to be updated we can decide to migrate to something else before it's late.

I was wondering the same thing.

fixed-data-table-2 seems to be much more actively maintained and what I'll be using on my project.

15 months ago after 1 month of starting to work on Frontend, seeing the state of frontend libraries repositories I concluded "Never use a JS library unless it's a big one with lots of stars in repo" because I saw many of them stop being maintained. Today I change that conclusion to "Never use a JS library unless it's ReactJS, babel-*, browserify, or things like that, or you don't expect any update on it". I go with implementing a table for my project as I don't know if fixed-data-table-2 will be maintained 2 months later.

Agreed, it is frustrating and wastes a lot of peoples time.

We can't blame the repository owner, he has his own life, but I don't know why it happens only to js repositories, not python repos for example. Maybe repo owners should spend some time and give others some responsibilities so that when they can't spend time, others can fill the gap.

Should we create a fork and start maintaining it somewhere else? What's the usual open source procedure in this case?
@glittershark Are you open to transferring ownership?

@Kadrian forking and maintaining elsewhere is usually not an appropriate decision unless repository owner doesn't respond for a long time. Usually the appropriate thing is that repository owner gives permissions to some other individuals who are interested if he doesn't have time to maintain himself.
I hope @glittershark can manage it.

@glittershark Any word on this?

Bueller...

Well i thought it would be good to make a fork of this and i added some features
Feel free to contribute https://github.com/elis-k/reactablex


Pending transaction due to low balance in VRF

why Vrf is getting too many links?

Looks like you'd need to double the amount of LINK in your account.


Update bundled PostgreSQL to 17.2

This also includes the ALTER TABLE rewrite optimization for domains
with constraints, see edgedb/postgres#3

Do I understand correctly that we will be shipping a modified version of postgres, but edgedb would still be compatible with the official builds of postgres?

Yes.


Entire server crash after few hours

Hello, I installed the last image on two differents server and after 10-15 hours, the two server crashed not just docker the entire server.
I tried again and I had the same issue next day.
Do you know if there is something wrong on the last release ?
And do you know how can I backup world/user achievements from de SSH of my server ?
OS: Debian 8.11
Server Management : Plesk
Deamon.log for dockerd : https://pastebin.com/J19TEN1W
Thank you in advance

Can you clarify what you mean by the “server”? Do you mean the Minecraft server container or the entire host running Docker?
I’m not sure if I can help much if you’re having dockerd issues.
As for backups, I do provide this image as one solution
https://hub.docker.com/r/itzg/mc-backup

I tried to host the minecraft image on two servers and each time after few hours the entire server crashed. Not just docker.

I do have the same problem. I'm running a PaperMc 1.15.2 Server. After one day the mc-server starts to lag and after few minutes the entire Docker host does not respond (cannot connect via SSH, other Docker Container stop working, etc..). I did start Papermc with --noconsole
This is my docker-compose file:
version: '3'

services:
  minecraft:
    image: itzg/minecraft-server
    volumes:
      - ./data/minecraft:/data
    environment:
      TYPE: "PAPER"
      EULA: "TRUE"
      MEMORY: "4G"
      SERVER_NAME: "Server"
      MOTD: "Super Duper Server"
      ENABLE_RCON: "TRUE"
      RCON_PASSWORD: "passwd"
      RCON_PORT: 28016
    restart: always
    ports:
      - 28016:28016
    command: --noconsole
  router:
    restart: always
    image: itzg/mc-router
    ports:
      - 25565:25565
    command: --mapping=mc.url.example=minecraft:25565

@micschwarz unfortunately it's really hard to diagnose these cases where the entire host is affected. The Minecraft server is just a java process so there's limited "damage" it can do other than resource starvation.
Here is some info to gather and things to try:

Make sure to docker pull the latest itzg/mc-router. There was a resource leak in an older version that caused high CPU usage
What does docker version report
What is your host platform
What does docker stats report as the system becomes heavily loaded
What does top report as the system becomes heavily loaded

I am using the latest Version of mc-router and minecraft-server
docker version report:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:29:52 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:28:22 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

I‘m using Ubuntu Server 18.4 LTS on a 8-Core 30GB RAM Virtual Server
I can‘t execute docker stats as well as htop, because I cant SSH on my Server when the load is high

I switched the Server to Java 13 and installt some monitoring. Maybe this will bring up some more information.

So I've been playing for about an hour now and after problems occured I've had a look at my Monitoring.

At ~15:20 I started to play. At ~16:00 lag started and the server kicked me.
At 16:00 the RAM usage reached 4,5GB (max allocated 4GB). I dont have any plugins installed, so 4GB Ram should be plenty for a maximum of 3 players online (but this time I was alone). This might be an issue with paper and javas gc

@micschwarz  Wow, I run a vanilla 1.15.2 server and stays well within the 2GB limit of the VM it runs on. I would recommend reporting an issue with the Paper team since that is indeed an usual memory behavior. You can do a ps -ef on the host to see what java command line is being used to run their server jar.


For sysinfo, show more memory information

The memory and swap information is displayed in the lower right corner.
For swap, it reveals the percentage in use based on the used value divided by toal
For memory, it reveals the percentage in use based on the used value divided by total.
However, for base memory, there is additional information that is useful to convey.  The memory that is buffer/cache is quasi-used, leaving the actual free amount much less.
Consider this source information
 free --mebi
              total        used        free      shared  buff/cache   available
Mem:           3888        1631         339         165        1917        1970
Swap:          2047         340        1707

Recommend enhancing the display to show the buff/cache value as dithered.  The goal is to show that the system is effectively using the memory it has available.

Nah. Sysinfo is fine with the memory as depicted. If anything, may reduce to one tile leaving space for network tx/rx


Jackett Support?

Hey there, I was wondering if this supports Jackett. It's similar to Prowlarr as it's an indexer.
This app looks great though.

Right now only Prowler is supported, as it's the one I'm using.
Will look into the API documentation for Jackett to see if it is easy to add

Looking further into it, seems like Jackett doesn't support searching only for freeleech torrents https://github.com/Jackett/Jackett/issues/9456
I also don't see any marker to see if a torrent is freeleech or not to manually filter them in yoink.

A little late, but just saw this. Almost all of our (Jackett's) indexers support freeleech search - https://github.com/Jackett/Jackett/issues/9456#issuecomment-689146324
Almost all of our indexers are the same as Prowlarr's (we update, they merge across), so if they support something then chances are so do we.


Renovate/postcss preset env 10.x

- Summary

- Test plan

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

👷 Deploy Preview for fluffy-sawine-f018e5 processing.

Name
Link

🔨 Latest commit
bb066f9f328376a159940c9b1f4d481755729382

🔍 Latest deploy log
https://app.netlify.com/sites/fluffy-sawine-f018e5/deploys/6731b8ad672e410008917155


To check: are tasks logs written at the end of tasks or during execution?

Relevant piece of code (from  _call_command_wrapper, in runnner/_common.py):
    fp_stdout = open(stdout, "w")
    fp_stderr = open(stderr, "w")
    try:
        result = subprocess.run(  # nosec
            shlex_split(cmd),
            stderr=fp_stderr,
            stdout=fp_stdout,
        )
    except Exception as e:
        raise e
    finally:
        fp_stdout.close()
        fp_stderr.close()

Docs for open (https://docs.python.org/3/library/functions.html#open) say:

buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size in bytes of a fixed-size chunk buffer. Note that specifying a buffer size this way applies for binary buffered I/O, but TextIOWrapper (i.e., files opened with mode='r+') would have another buffering. To disable buffering in TextIOWrapper, consider using the write_through flag for io.TextIOWrapper.reconfigure(). When no buffering argument is given, the default buffering policy works as follows:

Binary files are buffered in fixed-size chunks; the size of the buffer is chosen using a heuristic trying to determine the underlying device’s “block size” and falling back on io.DEFAULT_BUFFER_SIZE. On many systems, the buffer will typically be 4096 or 8192 bytes long.

“Interactive” text files (files for which isatty() returns True) use line buffering. Other text files use the policy described above for binary files.

Thus I expect buffering to be active by default, with something like 4KB or 8KB buffer size. Some log files are smaller than that, and then they likely will only be written at the end. Larger log files (e.g. for cellpose) should be written in chunks during execution:
1.4K 0.err
   0 0.out
 989 1_par_20200812-CardiomyocyteDifferentiation14-Cycle1_zarr_B_03_0_.err
   0 1_par_20200812-CardiomyocyteDifferentiation14-Cycle1_zarr_B_03_0_.out
1.9K 2.err
   0 2.out
 525 3_par_20200812-CardiomyocyteDifferentiation14-Cycle1_mip_zarr_B_03_0_.err
   0 3_par_20200812-CardiomyocyteDifferentiation14-Cycle1_mip_zarr_B_03_0_.out
 11K 4_par_20200812-CardiomyocyteDifferentiation14-Cycle1_mip_zarr_B_03_0_.err
   0 4_par_20200812-CardiomyocyteDifferentiation14-Cycle1_mip_zarr_B_03_0_.out
3.5K 5_par_20200812-CardiomyocyteDifferentiation14-Cycle1_mip_zarr_B_03_0_.err
   0 5_par_20200812-CardiomyocyteDifferentiation14-Cycle1_mip_zarr_B_03_0_.out

TLDR
We are probably buffering log files at 4K/8K buffer sizes, which means e.g. 100 log lines of 81 characters.
We can probably easily switch to single-line buffering (with buffering=1), but I don't know how to make it e.g. 1K (see example below).

import io
import matplotlib.pyplot as plt

single_log_line = "Log line: this is a typical log line, with some contents and some more contents.\n"
lines = list(range(500))

for buffering in [-1, 1, 10]:
    print(f"{buffering=}")
    path = f"/tmp/xxx_{buffering=}"

    fp = open(path, "w", buffering=buffering)

    file_size = []
    for i in lines:
        fp.write(single_log_line)
        with open(path, "r") as fp2:
            file_size.append(len(fp2.read()))
    fp.close()
    plt.plot(lines, file_size, label=f"{buffering=}", alpha=0.8, lw=2)

plt.title(f"{io.DEFAULT_BUFFER_SIZE=}, {len(single_log_line)=}")
plt.yticks(range(0, 40000, 8192))
plt.xlabel("Number of log lines emitted")
plt.ylabel("File size in bytes")
plt.legend()
plt.savefig("fig.png", dpi=256)

I'm closing this, as the buffering mechanism seems quite clear, and possible updates are defined.
Let's re-open if we think that we need to update our current 4K/8K buffering for logs.


METRON-746: Build Custom Checkstyle and IDE formatting settings

Contributor Comments
Pretty much just editing the POM to actually set up Checkstyle with the Google Code Style, per the dev thread.
See Google Java Style Guide
Note that the Checkstyle version is set to 7.7.  I've tested it with the plugin version and had no issues at all, but if anyone is concerned, we can lower it to either the default or another version.
IntelliJ warnings
To pull into IntelliJ, please see https://github.com/jshiell/checkstyle-idea#configuration
Essentially the instructions are to:

Install the plugin (Available in IntelliJ's directly)
Go into Settings -> Other Settings -> Checkstyle
Change "Checkstyle version" to 7.7.
Apply.  Otherwise the new file won't match the version and an error will be thrown.
Add a new Checkstyle file

This can either be a local file or a remote one (remote will attempt to update periodically, so if you don't want to repeatedly connect to the URL, download the file somewhere and load it in).  Either works. For 7.7, use google_checks.xml
If you have errors, you most likely need to make sure you're set to the right version and have applied it.

Select the checkbox for the new style
Apply

To ensure it's setup properly, open a file, e.g. metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
New warnings should show up that are based on Checkstyle, e.g.
Checkstyle: WhitespaceAround: 'if' is not followed by whitespace. Empty blocks may only be represented as {} when not part of a multi-block statement

Code formatting
There are two options for code formatting. One is to import the google_checks.xml file from above.  The other is to import Google's IntelliJ editor settings for coverage.  In practice, Google's direct IntelliJ settings seem to handle autoformatting better. The instructions are essentially the same for each.
Google's setup is described at https://github.com/HPI-Information-Systems/Metanome/wiki/Installing-the-google-styleguide-settings-in-intellij-and-eclipse.  The direct Checkstyle can be imported similarly, but choosing the CheckStyle Configuration when doing Import Scheme.
Required Follow-up
This will require updating the wiki at https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines (Section 2.2).  It needs to be switched to the Google code conventions, and the instructions above for formatting and warnings should be pulled in.
Testing
I still need to spin up full-dev, but to get the checkstyle report do:
mvn clean install -DskipTests
mvn site site:stage-deploy site:deploy -Dmaven.javadoc.skip=true

Navigate to file:///tmp/metron/site/checkstyle-aggregate.html in a browser.
You should see
The following document contains the results of Checkstyle 7.7 with google_checks.xml ruleset. and a set of warnings. In practice, most of them are fixed by batch autoformatting, per discussions, which will be a followup task.
Misc
No enforcement of Checkstyle is made, and in fact, by default google_checks.xml sets everything to warning. The thought is that we'd need to pretty significantly reduce the number of warnings, and probably think through how we want to handle it.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
mvn -q clean integration-test install && build_utils/verify_licenses.sh

[ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

This appears to also work in Eclipse (similar deal: Install Checkstyle from Eclipse Marketplace, import GoogleStyle with the same instruction link as IntelliJ).
Having said that, Eclipse really dislikes some plugins in our POMs, and it seems like an Eclipse thing.  I haven't dug in further, and to be blunt, am not interested in doing so unless someone really, really wants to work with Eclipse.

Actually, that's a lie, autoformat works, but checkstyle doesn't highlight anything.  Well, actually, Eclipse won't highlight any errors, including outright compilation errors.  If someone has a working Eclipse environment, I can help setup checkstyle, but I'm not spending time debugging why Eclipse doesn't know how to import a cleanly cloned Maven project.

Resolved conflicts.
@mmiklavc I think you said on another PR that you'd seen some issues with the Google formatting?  Any update on that?

@justinleet nothing in particular - what I noticed was that auto-formatting would format me right into breaking the checkstyle checks. I'd like to figure out how to get the two working consistently. I would like to spend zero time, or even negative time if it were possible, on thinking about code formatting as I add new classes or update existing code. ⌘-alt-L all day long, imho.

@mmiklavc I didn't see any of that at all.  All I saw was it either fixing things, or in certain cases (e.g. commas starting newlines) that it just left things alone.
If there's any examples, or a particular file, I can take a look at it and see if I can find a disconnect.  I'll also take a look and see if I can find any examples where a checkstyle error from autoformat is produced (at least where there wasn't already one).

Justin, should we have a local copy of the check style xml in the git repo itself?  Then we could make changes... and they will be automatically setup?

Yeah, we can.  I just didn't bother for first pass.  It's just a matter of grabbing the file, dropping it in, and referencing it.  It's like two lines in the files.
I'll go ahead and pull in the base google checks for the version I pulled in.

+1.  This is a big step forward in getting all of this cleaned up.  I would like to see it land early in the post 0.4.0 timeline so we can work through any issues.

I removed the Apache header from the checkstyle.xml, because after research Checkstyle is LGPL and I don't even want to be in the questionable space of adding Apache headers to their config files (even though I've seen other projects do this).  Including this is fine since it's just in our build tools, not part of our releases.
To reflect this, an exception for checkstyle.xml on the RAT check has been added.
Let me know if this is the incorrect strategy for handling this, and I'll update appropriately.

@mmiklavc I still haven't found (significant) issues with autoformat + checkstyle, assuming both the checkstyle is setup and the autoformatting is setup (preferably with Google's IntelliJ settings).  Are you okay with merging this in, and filing follow-on tickets for any issues?  Or is there something in particular you'd like to see done here?

@mmiklavc Any response to the above? I'm also happy to take a look through what you set up and see if there's anything missing in the instructions and so on, if that would help.

Hey @justinleet, apologies for this slipping between the cracks. I'll run this again and see if there's anything specific I can find.

As a note, I moved the checkstyle version up to 8 and will update the instructions.
@mmiklavc This'll be relevant if you look through anything.  I don't know that it'll break on 7.7, because there weren't major changes to the file.

Can I +1 this a second time?

+1. I get why we didn't include the code formatting rules in the source tree like the checkstyle.xml (covers more than 1 IDE). But we should link to the setup instructions in our dev guide somewhere.


Fix react-hook related warnings

Closes #313

All warnings now cleaned up.

We added a little bit more tweaks to the choropleth map in a previous patch, once you're done with resolving the conflicts we're good to go :)

@jeremyphilemon rebased this branch on to master.


Langchain MRKL Agent not giving useful Final Answer

Discussed in https://github.com/hwchase17/langchain/discussions/7423

Originally posted by aju22 July  9, 2023
Here is the code I'm using for initializing a Zero Shot ReAct Agent with some tools for fetching relevant documents from a vector database:
chat_model = ChatOpenAI(
            model_name="gpt-3.5-turbo",
            temperature="0",
            openai_api_key=openai_api_key,
            streaming=True,
            # verbose=True)

llm_chain = LLMChain(llm=chat_model, prompt=prompt)

agent = ZeroShotAgent(llm_chain=llm_chain, tools=tools, verbose=True, handle_parsing_errors=True)
agent_chain = AgentExecutor.from_agent_and_tools(
    agent=agent, tools=tools, verbose=True, memory=memory
)

However when I query for a response.
query = "Can you explain a use case example of chain of thought prompting in detail?"
res = agent_chain(query)

This is the response I get back:
> Entering new  chain...
Thought: The question is asking for a detailed explanation of a use example of chain-of-thought prompting.
Action: Lookup from database
Action Input: "use example of chain-of-thought prompting"
Observation: Sure! Here's an example of chain-of-thought prompting:

Let's say we have a language model that is trained to solve math word problems. We want to use chain-of-thought prompting to improve its reasoning abilities.

The prompt consists of triples: input, chain of thought, output. For example:

Input: "John has 5 apples."
Chain of Thought: "If John gives 2 apples to Mary, how many apples does John have left?"
Output: "John has 3 apples left."

In this example, the chain of thought is a series of intermediate reasoning steps that lead to the final output. It helps the language model understand the problem and perform the necessary calculations.

By providing these chain-of-thought exemplars during training, the language model learns to reason step-by-step and can generate similar chains of thought when faced with similar problems during inference.

This approach of chain-of-thought prompting has been shown to improve the performance of language models on various reasoning tasks, including arithmetic, commonsense, and symbolic reasoning. It allows the models to decompose complex problems into manageable steps and allocate additional computation when needed.

Overall, chain-of-thought prompting enhances the reasoning abilities of large language models and helps them achieve state-of-the-art performance on challenging tasks.
Thought:I have provided a detailed explanation and example of chain-of-thought prompting.
Final Answer: Chain-of-thought prompting is a method used to improve the reasoning abilities of large language models by providing demonstrations of chain-of-thought reasoning as exemplars in prompting. It involves breaking down multi-step problems into manageable intermediate steps, leading to more effective reasoning and problem-solving. An example of chain-of-thought prompting is providing a language model with a math word problem prompt consisting of an input, chain of thought, and output. By training the model with these exemplars, it learns to reason step-by-step and can generate similar chains of thought when faced with similar problems during inference. This approach has been shown to enhance the performance of language models on various reasoning tasks.

> Finished chain.

As you can observe, The model has a very thorough and exact answer in it's observation. However in the next thought, the model thinks it is done providing a detailed explanation and example to the human. So the final answer is just some basic information, not really answering the question in necessary detail.
I feel like somewhere in the intermediate steps, the agent thinks it has already answered to the human, and hence just does not bother to give that as the final answer.
Can someone please help me figure out, how can I make the model output it's observation as the final answer. Or to stop making the model assume it has already answered the question to the human.
Will playing around with the prompt template work?

Okay that may work. However I keep getting OutputParserException: Could not parse LLM output.
I have tried setting handle_parsing_errors=True as well as handle_parsing_errors="Check your output and make sure it conforms!", and yet most of the times I find myself getting the OutputParserException.
Here is an example of the error:
> Entering new  chain...
Thought: The question is asking for a detailed explanation of a use example of chain of thought prompting. I should first check if there is a clear answer in the database.

Action: Lookup from database
Action Input: "use example of chain of thought prompting"

Observation: Sure! Here's an example of chain-of-thought prompting:

Let's say we have a language model that needs to solve a math word problem. The problem is: "John has 5 apples. He gives 2 apples to Mary. How many apples does John have now?"

With chain-of-thought prompting, we provide the model with a prompt that consists of triples: input, chain of thought, output. In this case, the prompt could be:

Input: "John has 5 apples. He gives 2 apples to Mary."
Chain of Thought: "To solve this problem, we need to subtract the number of apples John gave to Mary from the total number of apples John had."
Output: "John now has 3 apples."

By providing the model with this chain of thought, we guide it through the reasoning process step-by-step. The model can then generate the correct answer by following the provided chain of thought.

This approach of chain-of-thought prompting helps the language model to decompose multi-step problems into intermediate steps, allowing for better reasoning and problem-solving abilities.
Thought:
---------------------------------------------------------------------------
OutputParserException                     Traceback (most recent call last)
[<ipython-input-76-951eb95eb01c>](https://localhost:8080/#) in <cell line: 2>()
      1 query = "Can you explain a use example of chain of thought prompting in detail?"
----> 2 res = agent_chain(query)

6 frames
[/usr/local/lib/python3.10/dist-packages/langchain/agents/mrkl/output_parser.py](https://localhost:8080/#) in parse(self, text)
     40
     41         if not re.search(r"Action\s*\d*\s*:[\s]*(.*?)", text, re.DOTALL):
---> 42             raise OutputParserException(
     43                 f"Could not parse LLM output: `{text}`",
     44                 observation="Invalid Format: Missing 'Action:' after 'Thought:'",

OutputParserException: Could not parse LLM output: `I have found a clear answer in the database that explains a use example of chain of thought prompting.`

Is there any other way in which I can mitigate this problem to get consistent outputs? Is there a way to use Retry Parser for this agent, if yes how?

@aju22 hey, I hope you have figured out how to solve this issue since I'm also dealing with it.
I honestly didn't fully understand the solution that was given by the bot, so would you please share your solution?

@Khalidzo For my particular use case, I wanted the model to act sort of a like a teacher and give bullet point answers for easy understanding.
With a bit of hacks and "prompt engineering", I get fairly good answers and also worked out a way around most frustrating LLM Output Parse Errors, and it works 99% of the time.
Please check out the final implementation in my project at: DocumentGPT
You will find the agent declared in def get_agent(self)
P.S: It does consume a good amount of tokens to get it working.

Great job! @aju22 This can effectively extract information from the Observations.


Description of Web3 development

Over the past two months, we have been gathering technical knowledge about web3. Unfortunately, there are very few rust demos available for Web3 reference. We have to spend a lot of time doing this, which delays the current optimization of web2. We think it is a bit unreasonable. We will temporarily remove crates that are not used in the next two months and then remove modules related to web3.
We want to gather enough knowledge to complete the current summer boot web2 framework before doing Web3 work, when we will create a new warehouse for development, which is challenging work and will always be open for discussion.

new repo summer web3


Manage /etc/postfix/virtual file (aliases)

This file records aliases for postfix users.
A default record should be given to be used as an administrator account.

Done in 962a98823e3b69a14c242eaac5fa3778c223e02f.


Verify that command receives specific data through stdin

Hi there !
I'm programmatically doing something like echo 0 */2 * * * my_user my_script | crontab - to add an entry in the current user's crontab.
Can I check that the crontab really got 0 */2 * * * my_user my_script from STDIN with bash_shell_mock ?

Sorry, but I don't understand what you mean with match it agains arguments. As far as I can tell, using a pipe is the only programmatic way to update the crontab

Currently the shellmock_verify does not provide a way for you to verify the contents of STDIN that was passed into your mock.    It only allows you to verify that the command was called and what the arguments were when it was called.
I do not think it would be too difficult to allow you to define a mock that accepts STDIN and then allow you to match the standard input some how in the verify.  Correlating the which stdin goes with which mock would be the most challenging part I think.
Does that make sense?

Hi and thanks for your answer !
I'll do my best to answer, although I've started to use shellmock last month : my knowledge of shellmock internals is still lacking I guess !
So, If I remember correctly, there is an .out file that shellmock produces in which the command invocationsgenerated by shellmock_expect  reside.
shellmock_verify parses this file and gives us an array of lines so that we can match our expectations.
I guess that a mocked command that would accept a value from STDIN would need to be created with an --stdin flag passed to shellmock_expect. Wouldn't it result in putting something like echo <value-of--stdin> | mocked-command in the .out file ?
Now, I don't see how different stdin values could be mismatched in that scenario

I myself have to go back and look at the code to see how i  implemented the mocks but that sounds about right.  I am going to work on this one today I think.  This one is easily solvable particularly if I require a --stdin as you suggested.  The only issue is " and ' which is already an issue in shellmock.  Because I am doing everything within the shell I am bound by the shell.  You can never match quotes unless they were escaped and part of the original string value. Otherwise the shell processes them and my scripting would never see them.

This is what the guts of a mock look like.
shellmock_replay is what looks for a matching response.  I think the the stdin values should be considered when looking up a response.  That way you could return a different response based on different inputs.  Currently only args are considered for matching.  The matching should include stdin plus the args I think.
#!/bin/bash
export BATS_TEST_DIRNAME="/Users/wut839/workspaces/opensource/bash_shell_mock/sample-bats"
. shellmock
shellmock_capture_cmd grep-stub "$*"
shellmock_replay grep "$*"
status=$?
if [ $status -ne 0 ]; then
    shellmock_capture_err $0 failed
    exit $status
fi

@CaptainQuirk
Can you possibly take a look at this solution to the stdin issue?
https://github.com/capitalone/bash_shell_mock/pull/28
I am out of vacation and don't want to push and break stuff, however, the PR is available for review.  I updated the readme as well in the shellmock_expect section.  I added two new arguments --match-stdin and --stdin-match-type which controls the input data and the rule used to perform the matching.
This PR was built on another PR so it also affects how arguments with spaces in them are matched and verified.  You can disable this feature -- assuming i did it right -- by defining SHELLMOCK_V1_COMPATIBILITY.

Thanks ! I'll have a look !

Hi @ckstettler ! Sorry for the delay about this ! I'm using bash_shell_mock inside a custom docker image I made with bats and some bats helper.
I can confirm that I managed to make my existing test suite work with the SHELLMOCK_V1_COMPATIBILITY flag set ! So you did it right !
But it also means I will have to update all uses of shellmock_expect to the new way of doing things ! bummer ! :smile:
I will try and get the stdin part tested later this week !

Non-compatibility is a bummer.   It seemed necessary to fix it properly however I am open to any suggestion.  Keep in mind if you don't feel that the quote material impacted your tests then you can define the flag globally on and then unset it in new tests and use the new format until a convenient time.
I could create a new expect function or make it a switch in there.

PR #26 resolves this issue.

This enhancement was added as part of release 1.3.

This enhancement was added as part of release 1.3.


puppetlabs-concat dependency version causes issues

We're experiencing this when using librarian-puppet:

Since there's no upper limit to the version (it only specifies >= 1.1.0), there are dependency resolution issues when another module sets >= 1.1.0 <3.0.0

We have a workaround for this for now, but it would be good to set this as a range regardless

Is this the error I'm getting here when trying to use 0.2.9?
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type concat::fragment at /etc/puppet/modules/consul_template/manifests/config.pp:17 on node <node>

@madAndroid - Apologies for the delay in getting to look at this, I've added proper bounds to the dependencies in metadata.json - what was your original issue?
@bmhkb4 - Your issue sounds like you don't have the concat module installed. I'd check this first, and raise a separate issue here if you do.


serve css from the current desk

... instead of hardcoding home. see #252

Seems like a good idea to me.  @ohAitch, want to weigh in?


Peter add code cov

All checks are not passed. Codecov is failing. Please check.

That's because we don't have code cover in the master branch. Please merge
it and next time it should work.
On Wed, Nov 8, 2017, 17:07 DB3619<EMAIL_ADDRESS>wrote:

All checks are not passed. Codecov is failing. Please check.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/nyu-devops-golf-team/wishlist/pull/38#issuecomment-342976953,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQeNHaEdx0-Enhc-G6nX1h-3_IVvcXEks5s0iYcgaJpZM4QKKQF
.

--
Thank you,
Yonghong Chen
Master of Computer Science
New York University
NYC, NY

ok. I am proceeding to merge.


Exclude JPype 0.7.2 from CI; add python3-setuptools for TeamCity

Required to un-break builds failing with messages like this.
The Dockerfile is also formatted identically to message_data.
How to review
Confirm the TeamCity CI check passes.
PR checklist

[x] ~Tests added.~ N/A; CI only.
[x] ~Documentation added.~ N/A
[x] ~Release notes updated.~ N/A

Note that a bug should be filed upstream with JPype to address the segfaults caused by 0.7.2.

better to use git mv (or similar from GUI tools) to be able to compare changes.

I did, but if the file has a high enough fraction of modified lines, then git (diff) views it as delete-and-create rather than rename-and-edit.

Conda first installs JPype1 0.7.2, and then the ixmp dependency forces it back down to 0.7.1. For some reason, only on AppVeyor/Python 3.7 (i.e. not 3.6), this caused errors like:
___ ERROR at setup of TestTimeSeries.test_remove_geodata[TimeSeries-single] ___
request = <SubRequest 'test_mp' for <Function test_init[TimeSeries]>>
tmp_env = environ({'7ZIP': '"C:\\Program Files\\7-Zip\\7z.exe"', 'ALLUSERSPROFILE': 'C:\\ProgramData', 'APPDATA': 'C:\\Users\\ap...T_CURRENT_TEST': 'ixmp/tests/core/test_timeseries.py::TestTimeSeries::test_remove_geodata[TimeSeries-single] (setup)'})
test_data_path = WindowsPath('C:/projects/ixmp/ixmp/tests/data')
    @pytest.fixture(scope='class')
    def test_mp(request, tmp_env, test_data_path):
        """An empty ixmp.Platform connected to a temporary, in-memory database."""
        # Long, unique name for the platform.


Create a new endpoint or extend the project mock API endpoint to serve information on an individual project

In addition to serving information on the matrix file(s) and related metadata within a project, the browser wishes to populate the information on the project programmatically. See this example. Once we know which fields are important to send to users (#24), we should create a mock endpoint (or extend #17) that the browser can develop against.
AC:

[x] Create a mock endpoint whose payload contains all the information needed to populate a basic project page.

Blocked by #24

mock complete


TypeError: Cannot read property 'cookieValue' of undefined from Renault-ze node

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to 'basic flow', add email and password
Click on 'inject'
Go to 'debug node'
See error: "TypeError: Cannot read property 'cookieValue' of undefined"

Expected behavior
data from API
Screenshots

Version (please complete the following information):

OS: win11
Browser: Chrome
Node-Red: 1.1.3

Additional context
If I go to "http://services.renault-ze.com/user/login" page not found!

maybe an issue with country / car type. for it it is just normal working


Correct starts_on error messages

Changes in this PR:

Noticed issues with some of the error messages for the starts_on field on the important dates form.
New error messages are from our content designer.

Review app deployed to https://teaching-vacancies-review-pr-3493.london.cloudapps.digital


OpossumReporter: Miscellanoues improvements

Besides a couple of style fixes this PR implements handling for scope excludes contained in the given ORT file.
As a consequence the option to specify the excluded scopes manually is dropped, for consistency.
There are a couple of further simplifications / refactoring which prepares for migrating from the legacy project scope to using the dependency navigator for accessing the scopes.
BREAKING CHANGE:
The option OpossumReporter.OPTION_EXCLUDED_SCOPES is removed. Users should stop passing this option to the cli.
As replacement for the functionality, scope excludes have to be defined in the ORT file which is input to the reporter.

/cc @maxhbr


What is the GPU mem requirement for finetuning?

We now support single gpu training now. However, using one GPU for full fine-tuning can be slow.

We now support single gpu training now. However, using one GPU for full fine-tuning can be slow.

can dreambooth finetuning be done on a consumer GPU?


Multimedia content guidance

A blog started by Andrew Arch: https://docs.google.com/document/d/1_m4e_TISTBNrMlSjDeHe10B6GyvVxD33N_QxcEgPKUE/edit?ts=597a9987 we may be able to repurpose for content guidance.

Now being tracked in Trello


[2023-07-23] Code Review

Dear Fajar Anggara,
Here is my review of your code. So far, there are still many things to fix up. Please read my comment on this PR for the full review. Keep up the good work, thank you.
Best Regards,
Ian

Dear sir Ian,
Thank you very much for your review on the program, I will repair the program as your suggestion. I will underline the important note  such as making program as clean as possible, do not forget to hide the console log, and using an object to handle identical variables.
Best regards Fajar


Time seek wraps around beginning and ending

It's awesome you included the feature I asked for, thanks! But to help improving your video player I want to report this issue
Given I'm playing a video
And time is at 0:05
When I press left arrow to go back in time
Then the time goes to the near-end of the video
Same behavior is shown when I go to forward near the end, I go back to the near-beginning

Hello @airabinovich,
Yes this is the intended effect.

Say the video is Y time long and going backwards once would put it at time 00:00 - X then the slider position is put at Y - X.
Would you rather it truncate to 00:00?
:+1:

Hi @lettier,
As I see it, you can accidentially jump to the near-end of a video and watch some part you shouldn't have watched yet (spoilers)
I'd truncate the time seek at 00:00 and at Yas well. (Being Y the video length)
On another matter, I believe the time seek step is a portion of Y. I'd rather make it a fixed length (10 seconds I think is a good span)

Hello @airabinovich,
I changed the functionality to now stop at the beginning when seeking left and to stop at the end when seeking right.
I also reduced the step size for a more granular experience.
You can try out the pre-release here.
:+1:

It looks good to me now, great!


Undo bad changes from f01ca07f Squash this giant branch (#66)

Codecov Report

Merging #67 (09b860b) into master (88b291d) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master       #67   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           29        29
  Lines         1412      1413    +1
  Branches       205       205
=========================================
+ Hits          1412      1413    +1

Impacted Files
Coverage Δ

src/rics/mapping/_heuristic_score.py
100.00% <100.00%> (ø)

src/rics/mapping/_mapper.py
100.00% <100.00%> (ø)

src/rics/mapping/score_functions.py
100.00% <100.00%> (ø)

src/rics/mapping/types.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 88b291d...09b860b. Read the comment docs.


relax elixir language requirement; fix warnings

Coverage remained the same at 100.0% when pulling 4fb152bc135407488c7ddb81f6678e99ba66f7a2 on ericsullivan:elixir_1_5_upgrade into e10f9455e21eac6de8c654c5777ba6fe65bb6169 on aditya7iyengar:master.


Move ProduceRequestTransformation

This is currently in the production proxy code. It should be a test class.

Triaged 1 Nov 2023 - not a priority, needs addressing. Not the only filter in prod code that needs moving. Used by other modules, so more complicated than it looks.

Done!


Interoperability issues between releases 1.1.1 and 1.3.x with hot WAL journal

The following scenario may seem somewhat strange or constructed, but it depicts a real situation given the sqlite3mc release 1.1.1 we have in the field (and due to #39 I'm even somewhat happy we didn't switch).
Overall scenario: one application A writes to a new database in WAL mode, data inserted in a transaction is already commited, then the application crashes or is killed, leaving a hot WAL journal behind (The fact that SQLite is normally able to incorporate that journal into the database when the database is opened the next time meets a desirable design goal in our case, i.e. providing fail-over safety). Another application B should then still be able to open the database with all the commited data being available. Eventually, it will then switch back to DELETE journal mode.
This scenario worked well with System.Data.SQLite (used before we switched to sqlite3mc). Note that RC4 encryption here is only a legacy thing, current databases use chacha20 but we need to provide compatibility.
Now, depending on the sqlite3mc versions in use for A and B we see different errors as described below.
I recreated the issue just using the appropriate versions of the sqlite3mc shell (binaries as provided in the github releases) using the scripts below. "master" means shell binary built from sources downloaded after commit 43bd436.
Database creation (using 1.1.1):
.open newtest.db
pragma cipher=rc4;
pragma rekey='123';
create table test (x int);
pragma journal_mode=wal;
begin immediate;
insert into test (x) values (1);
commit;
*** kill shell here ***

Reading database with hot WAL journal:
.open newtest.db
pragma cipher=rc4;
pragma key='123';
pragma journal_mode;

1.1.1 and 1.3.1: wal
master: delete
select * from test;
1.1.1 and 1.3.1: no data
master: Error: no such table: test
.open (close db)
1.1.1 and 1.3.1: wal journal discarded, data lost
master: wal journal still exists, but:
.open newtest.db
.dbinfo

master: unable to read database header
It's unclear to me if the database created with 1.1.1 is broken (but can be read with 1.1.1 with no problems) or 1.3.x stumbles or maybe both.

The following scenario may seem somewhat strange or constructed, but it depicts a real situation given the sqlite3mc release 1.1.1 we have in the field (and due to #39 I'm even somewhat happy we didn't switch).

 However, such things happen unfortunately - as even SQLite itself demonstrated recently with 5 bug fix releases in rapid succession (from version 3.35.0 to 3.35.5).

Overall scenario: one application A writes to a new database in WAL mode, data inserted in a transaction is already commited, then the application crashes or is killed, leaving a hot WAL journal behind (The fact that SQLite is normally able to incorporate that journal into the database when the database is opened the next time meets a desirable design goal in our case, i.e. providing fail-over safety).

Even without any application crash it is a typical situation that concurrently running applications access the WAL journal file to read the latest database modifications. And the changes introduced with SQLite3 Multiple Ciphers version 1.3.x add support to access databases in WAL journal mode from old legacy applications (like System.Data.SQLite) and new applications using SQLite3 Multiple Ciphers. However, this compatibility comes at a price: hot WAL journal files created by SQLite3 Multiple Ciphers versions 1.2.5 or below are not compatible with hot WAL journal files created by versions 1.3.0 or later.

Another application B should then still be able to open the database with all the commited data being available. Eventually, it will then switch back to DELETE journal mode.
This scenario worked well with System.Data.SQLite (used before we switched to sqlite3mc).

This is not surprising, because System.Data.SQLite used the (no longer existing) SQLITE_HAS_CODEC API for encryption support.

Now, depending on the sqlite3mc versions in use for A and B we see different errors as described below.
I recreated the issue just using the appropriate versions of the sqlite3mc shell (binaries as provided in the github releases) using the scripts below. "master" means shell binary built from sources downloaded after commit 43bd436.
Database creation (using 1.1.1):
Reading database with hot WAL journal:
1.1.1 and 1.3.1: wal
master: delete

Is this the reported journal mode?

select * from test;
1.1.1 and 1.3.1: no data
master: Error: no such table: test
.open (close db)
1.1.1 and 1.3.1: wal journal discarded, data lost
master: wal journal still exists, but:
.open newtest.db
.dbinfo

master: unable to read database header

All this is not really surprising, because the WAL journal file created by version 1.1.1 can't be accessed successfully from newer versions.

It's unclear to me if the database created with 1.1.1 is broken (but can be read with 1.1.1 with no problems) or 1.3.x stumbles or maybe both.

Versions from 1.0.0  up to 1.2.5 should be able to operate in WAL mode without problems. The WAL journal file is consistent across those versions. However, concurrent access with legacy applications will not work.
Version 1.3.0 introduced WAL journal mode compatibility with legacy applications, so that concurrent access with legacy applications works (at least it would if bug #39 did not exist). However, concurrent access with versions 1.2.5 and below will unfortunately not work due to incompatible WAL journal files.
Unfortunately there exists no "fix" for this problem. It is unlikely that I revert the WAL journal file format to that of version 1.2.5 (or below), because in my opinion compatibility with legacy applications in WAL mode is more important than compatibility with prior versions of SQLite3 Multiple Ciphers.
For a transitions from version 1.2.5 or below to version 1.3.x it will be necessary to perform a full WAL checkpoint operation, for example with
PRAGMA wal_checkpoint(TRUNCATE);
This guarantees that the WAL file is empty, and the database file itself will be accessed.

I decided to make the WAL compatibility configuration setting publicly available. Commit b8e0cd7d581291684fd1248aa5ab001d6b15b4fc provides the adjusted code. Please note that I renamed the parameter to mc_legacy_wal. This parameter may be set via pragma or via URI parameter.
Please also note that the parameter setting can't be changed after the database connection has been established. That is, the sequence of pragmas should be as follows:
-- Select cipher scheme
PRAGMA cipher=chacha20;
-- Set WAL compatibility with versions <= 1.2.5
PRAGMA mc_legacy_wal=1;
-- Set passphrase
PRAGMA key='passphrase';
-- Change journal mode to recover hot WAL journal
PRAGMA journal_mode=delete;

Thereafter the database connection should be closed. On reopening the database connection do not set mc_legacy_wal (that is, use implicitly mc_legacy_wal=0).

Thanks for investigating further.
Do you think it would be possible to make sqlite3mc fail-safe in the regard that on opening without the new pragma, it just fails on hot journals from <=1.2.5 consistently without deleting the WAL?
I.e. before this happens:

SQLite tries to recover the WAL journal as it should, but fails due to incorrect WAL frame checksums. However, SQLite simply ignores invalid WAL frames and resets the WAL journal thereafter.

Do you think it would be possible to make sqlite3mc fail-safe in the regard that on opening without the new pragma, it just fails on hot journals from <=1.2.5 consistently without deleting the WAL?

Not without patching SQLite itself.

I.e. before this happens:

SQLite tries to recover the WAL journal as it should, but fails due to incorrect WAL frame checksums. However, SQLite simply ignores invalid WAL frames and resets the WAL journal thereafter.

The problem is that from SQLite's perspective it is totally normal to come across "invalid" WAL frames (which are then silently ignored), because WAL frames can be reused and may not belong to the current chain of frames. On opening a database SQLite checks if the database is in WAL mode, and if this is the case, SQLite checks whether a WAL file exists. If yes, SQLite tries to recover whatever is in the WAL journal. After that operation the WAL journal is reset to force a consistent WAL state.

Release 1.3.2 fixes the underlying problems and provides a configuration parameter to facilitate the transition from prior versions. The documentation has been updated accordingly. Therefore this issue will be closed. Reopen it if necessary.


[ROCM] Fix gpu_atomic_test for ROCm

Fixes test for ROCm platform

Can we unify two branches?

@cheshire  Sorry, I am not following. Can you please explain which branches to unify?

Hi @cheshire Can you please review this PR ? Thank you!


Add build.OperationBuilder interface

Consider a following code:
var operationBuilder interface{}
if !destinationExists {
	operationBuilder = b.CreateAccount(mutators...)
} else {
	operationBuilder = b.Payment(mutators...)
}

// Check errors here

txBuilder := b.Transaction(
	operationBuilder.(b.TransactionMutator)),
)

If we wanted to check if there were errors after applying mutators (Err field in builders) it's not possible because type assertion works for interfaces only. To solve this we can create an interface:
type OperationBuilder interface {
	MutateTransaction(o *TransactionBuilder) error
	Mutate(muts ...interface{})
	Error() error
}

Really good point.  Let me think about this some, since I'm not sure if an interface is specifically how we want to handle it...  particularly I'm shying away from using Error() since that method name is used for the builtin error interface (and it returns a string).


Rich text editor control - table support

Category
[x] Enhancement
[ ] Bug
[ ] Question
Version
I saw in a previous thread from @hugoabernier that it may be possible to extend this to have the ability to insert tables, as per the out of the box editor.
It would be great to have that functionality.

I found this post.
With a little rework this looks like a good solution (he basically copied the current modern approach):
http://damjan.blogspot.com/2020/04/sharepoint-online-use-classic-rich-text.html


mean and sky-subtracted images for a particular SDSS frame

Given a particular SDSS frame/image, generate a new image with the same format and metadata as the SDSS one, but, depending on a command-line argument, where each pixel's value is 1) mean brightness of the variational distribution (in nanomaggies), or 2) sky subtracted brightness (Celeste minus SDSS).
For a particular frame, it's first necessary to load the relevant part of the Celeste catalog, stored in
/global/cscratch1/sd/jregier/celeste_sdds_catalog_jld

The file names of the jld files in that directory indicate what region of the sky they describe.
There's some code here that shows how to render images given variational parameters. It's for an older version of Celeste though (the fft branch), so the syntax would need to modified slightly.
We'd call the script created for this issue with taskfarmer, to run it on every frame of SDSS. We'd upload the renderings to legacysurvey.org. (issue #288).

@gostevehoward Does this issue make sense? Would you work on it when you have time?

I think it makes sense. What is a frame, exactly?

Let me quote the SDSS glossary so I'm not slightly off:
Frame: The data stream from a single CCD in a scanline is cut into a series of frames which measure 2048 × 1489 pixels and overlap 10% with the adjacent frame. The frames in the 5 filters for the same part of the sky are called a field.

Basically, frame = Image. Or at least there's a 1-1 mapping between SDSS frames them Celeste Images, and they contain mostly (entirely?) the same data. 5 frames per field (in SDSS). 5 images per RCF (in Celeste).

I see, so RCF + band specifies a frame, then?

Yes, that's it.

@jeff-regier So this script should take an RCF and band, e.g., 325/5/110 "r", and then

download the frame FITS from the SDSS server, e.g., frame-r-003325-5-0110.fits
load the corresponding Celeste catalog(s) from celeste-*.jld,
generate a new FITS containing the same four extensions as the SDSS frame file, where everything is the same except the pixel values in the first extension,
and generate the mean brightness or sky-subtracted brightness values using the catalog?

In the catalog filenames, what's the difference between celeste-A-B-C-D.jld and celeste-A-B--C--D.jld? Is A-B a RA range and C-D a dec range, in the former case?
When you say "sky-subtracted brightness (Celeste minus SDSS)", do you intend mean brightness from the variational distribution minus SDSS sky level? With the output still in nMgy, I assume?
Thanks for the help, want to make sure I get it right.

Yes, that all sounds right. It may be better to generate the whole field (all 5 frames) at once for a particular RCF, rather than just one frame. So the user specifies 325 5 110 on the command line but not "r".
To download the SDSS data, use the makefile in test/data. You can just download everything for the field. (Includes all 5 frames, and some other stuff.)
I think there's only the celeste-A-B-C-D.jld format. If you see two hyphens in a row, the second one is probably a minus sign. :-) Some of the declinations are below the equator.

Got it, thanks :)

I'm failing to find catalog files for some sky regions. Here's the field_extents.fits row for RCF 3325/5/110:
│ Row │ run  │ camcol │ field │ ramin    │ ramax    │ decmin   │ decmax   │
├─────┼──────┼────────┼───────┼──────────┼──────────┼──────────┼──────────┤
│ 1   │ 3325 │ 0x05   │ 110   │ 0.254468 │ 0.418259 │ 0.407708 │ 0.632983 │

This would require the files celeste-0.2500-0.5000-0.2500-0.5000.jld and celeste-0.2500-0.5000-0.5000-0.7500.jld.

The "missing" box may actually be multiple boxes. (The boxes aren't all 0.25 x 0.25.)
$ ls $SCRATCH/celeste_sdds_catalog_jld/*celeste-0.[234]*-0.[2345]*-0.[567]*-0.[567]*.jld
/global/cscratch1/sd/jregier/celeste_sdds_catalog_jld/celeste-0.2500-0.3750-0.5000-0.6250.jld
/global/cscratch1/sd/jregier/celeste_sdds_catalog_jld/celeste-0.2500-0.3750-0.6250-0.7500.jld
/global/cscratch1/sd/jregier/celeste_sdds_catalog_jld/celeste-0.2500-0.5000--0.7500--0.5000.jld
/global/cscratch1/sd/jregier/celeste_sdds_catalog_jld/celeste-0.3750-0.5000-0.5000-0.6250.jld
/global/cscratch1/sd/jregier/celeste_sdds_catalog_jld/celeste-0.3750-0.5000-0.6250-0.7500.jld

Ah ha, thanks.

This is coming along (code here), here's a side-by-side of a noiseless expectations image (left) and a true frame (right):

There are a lot of these weird bright galaxies in the expectations image with very lower minor-to-major axis ratios. They do seem to be in the catalog: check out this CSV version of the corresponding catalog, where row 404 (highlighted blue in the spreadsheet) seems to correspond to the galaxy under my mouse (in the left window). Half-light radius of 21 and flux of 490.

That is weird. It could be that the catalog was generated with an old version of Celeste. Would you mind adding a flag to the script that causes it to regenerates the relevant part of the Celeste catalog rather than uses the (potentially stale) Celeste catalog? Feel free to decline if you're out of time. I believe it's just a matter of calling one_node_joint_infer after invoking the makefile in test/data enough times to load the SDSS data locally. It'd be generally useful to be able to run your script without having a copy of the Celeste SDSS catalog on hand.

No problem, benchmark/accuracy/run_celeste_on_field.jl already does that. I'll have to do a little file format finagling but shouldn't be a big deal.

Seems to be working reasonably well now:

This is from Celeste initialized with coadd data:
Celeste.jl/benchmark/accuracy$ julia write_ground_truth_catalog_csv.jl coadd ../../test/data/coadd_for_3325_5_110.fit
Celeste.jl/benchmark/accuracy$ julia run_celeste_on_field.jl --use-full-initialization output/coadd_for_3325_5_110_f86460183f.csv --run 3325 --camcol 5 --field 110
Celeste.jl/benchmark/accuracy$ julia ../../bin/write_celeste_expectation.jl 3325 5 110 --catalog-dir output [--subtract-sdss]

If you wanted Celeste initialized with primary data, substitute these first two commands:
Celeste.jl/benchmark/accuracy$ julia sdss_rcf_to_csv.jl --run 3325 --camcol 5 --field
110
Celeste.jl/benchmark/accuracy$ julia run_celeste_on_field.jl --use-full-initialization
 output/sdss_3325_5_110_primary_df2758e55d.csv --run 3325 --camcol 5 --field 110

Here is Celeste expectation minus SDSS value:

Very nice, Steve. That's pretty much what we'd expect the sky-subtracted images to look like.
Would you add documentation to the wiki for this script too,if you haven't already?

Documented here: https://github.com/jeff-regier/Celeste.jl/wiki/Rendering-Celeste-posterior-expectations


Dont use ternary

ternary is not available in ansible 1.7, which is used (at least on jessie) when the ansible_local provisioner is used.
Replace them with 'foo' if bar else 'baz'
also cast php_version_installed to float where needed, this makes #53 unnecessary  (it was used in just one line which was not a ternary anyway)

Good to know about the ansible version and thanks for the fix !
If the need arise to have ansible > 1.7 I think we can have it by installing the jessie-backports via a shell provisionner before asking vagrant to install ansible, or maybe just install the ansible version we want ourselves in a shell provisionner, or update the base boxes.