Webdataset (Liaon115M) + Torchlightning (pl.DataModule) with visualizing progressbar during training

This issue aims to end with the discussion that how to run the webdataset with torchlightning.
Several concepts should be clarified here :

If we use webdataset, it means the dataset is large-scale (if not, then why bother ?), so the multi-gpu support is definitely needed!!
Yes, dataset length is impotent. No one! yes, no one want to train on a dataset without knowing the step per epoch in large-scale dataset!!. So, dataset length (num of data sample in dataset) is definitely needed!!
It's weird that torch IterableDataset doesn't support too much in multi-gpu manner. Because the reason we use IterableDataset just exactly that we're working with the large-scale dataset and Multi-gpu is necessary for large-scale training.

ok, if dataset length is impotent, then how can I get it ?

while, we wrote a parallel script for scanning the large-scale dataset, and record the image for each tar-file and store in a json file with dict manner : {'/data/laion115m/00000.tar':1147, '/data/laion115m/00001.tar':..., /data/laion115m/11164.tar' : 551}.

After some murmur, let's begin 👍

First step, use Webdataset with toy usage only (for-loop with single stream iteration)

# feed dummy pipe `wds.split_by_worker`, which is the default of workersplit, thus it'll not affect the results
dataset = wds.WebDataset(url, nodesplitter=wds.split_by_worker).shuffle(1000).decode('pilrgb', handler=wds.warn_and_continue).to_tuple("jpg", "txt", handler=wds.warn_and_continue).map(trfs)

# toy usage :
for batch_id, sample in enumerate(dataset):
      print('ok, good! we can get the sample')

Wrap webdataset in torch IterableDataset (aims to do nodesplit)

from torch.distributed import get_rank, get_world_size

class Iter_ds(torch.utils.data.IterableDataset):
    def __init__(self, urls, trfs, n_sample):
        self.urls = urls
        self.trfs = trfs
        self.n_sample = n_sample

    def __len__(self):
        # let's say i have 100 image totally, 2 gpus, batch_size = 4.
        # then n_step per epoch should be : `100 // (2 * 4) = 17`, and last batch doesn't fill with 4 samples.
        # in here, we directly control how many n_step by __len__.
        # for the above example, in here should be `100 // 2`, then torch dataloader will try to divided this number by batch_size automatically ~ (since we setup batch_size in torch dataloader)
        return self.n_sample // get_world_size()

    def __iter__(self):
        process_rank = get_rank()
        world_size = get_world_size()
        for url in self.urls:
            # feed dummy pipe `wds.split_by_worker`, which is the default of workersplit, thus it'll not affect the results
            # or the default `wds.single_node_split` will broken the node_split procedure
            dataset = wds.WebDataset(url, nodesplitter=wds.split_by_worker).shuffle(1000).decode('pilrgb', handler=wds.warn_and_continue).to_tuple("jpg", "txt", handler=wds.warn_and_continue).map(self.trfs)
            for batch_id, sample in enumerate(dataset):
                # assign a independent batch for the gpu wrt. gpu_id (nodesplitter in here)
                if batch_id % world_size == process_rank:
                    yield sample
                # skip the batch it doesn't belong to the gpu
                else:
                    continue

The last step, take torch IterableDataset as a member in pl.DataModule

class Liaon115M(pl.LightningDataModule):

    def __init__(self,
        data_dir: str = "path/to/dir",
        split_ratio: list = [0.7, 0.1, 0.2],
        img_transforms = None,
        txt_transforms = None,
        num_workers: int = 4,
        batch_size: int = 16,
        num_epoch: int = 1,
        pin_memory: bool = False
    ):
        super().__init__()
        self.data_dir = Path(data_dir)
        self.split_ratio = split_ratio
        self.img_transforms = img_transforms
        self.txt_transforms = txt_transforms

        self.trfs = lambda tup : (self.img_transforms(tup[0]), self.txt_transforms(tup[1]))

        self.batch_size = batch_size
        self.num_epoch = num_epoch
        self.num_workers = num_workers
        self.pin_memory = pin_memory

        # laion 115M, success downloaded 104881705 records..

    # chking dataset is downloaded..
    def prepare_data(self):
        with self.data_dir.open('r') as js_ptr:
            self.tar_dict = json.load(js_ptr)

        tar_lst = list(self.tar_dict.keys())
        n_shard = len(tar_lst)

        tra_ratio, val_ratio, _ = self.split_ratio
        self.tra_lst = tra_lst = tar_lst[ : int(n_shard * tra_ratio) ]
        self.val_lst = val_lst = tar_lst[ len(tra_lst) : len(tra_lst) + int(n_shard * val_ratio) ]
        self.tst_lst = tar_lst[ len(tra_lst) + len(val_lst) :  ]

    def _get_sample_num(self, tar_lst):
            cnt = 0
            for tar_key in tar_lst:
                cnt += self.tar_dict[tar_key]
            return cnt

    def setup(self, stage: str = 'train', rank=None, world_size=1):
        self.prepare_data()
        if stage == 'train':
            n_tra_sample = self._get_sample_num(self.tra_lst)
            print(f'Total sample in train ds : {n_tra_sample}')
            self.laion_train = Iter_ds(self.tra_lst, trfs=self.trfs, n_sample=n_tra_sample)
            n_val_sample = self._get_sample_num(self.val_lst)
            print(f'Total sample in valid ds : {n_val_sample}')
            self.laion_valid = Iter_ds(self.val_lst, trfs=self.trfs, n_sample=n_val_sample)
        else:
            n_tst_sample = self._get_sample_num(self.tst_lst)
            print(f'Total sample in test ds : {n_tst_sample}')
            self.laion_test = Iter_ds(self.tst_lst, trfs=self.trfs, n_sample=n_tst_sample)

    def train_dataloader(self):
        loader = torch.utils.data.DataLoader(
            self.laion_train,
            batch_size=self.batch_size,
            shuffle=False,
            pin_memory=True,
            num_workers=self.num_workers,
            prefetch_factor=2,
            drop_last=True
        )
        return loader

    def val_dataloader(self):
        loader = torch.utils.data.DataLoader(
            self.laion_valid,
            batch_size=self.batch_size,
            shuffle=False,
            pin_memory=True,
            num_workers=self.num_workers,
            prefetch_factor=2,
            drop_last=True
        )
        return loader

Since we setup the len in iterableDataset, it will show the n_step and epoch in progress-bar

version of package 🥇

webdataset : 0.2.86
pytorch-lightning : 2.2.1
torch : 2.2.0+cu118

ok, now it's peace, no argument for torchlightning + webdataset, hopefully...

HAPPY CODING WITH TORCHLIGHTNING!!

Hi, Thanks for your code. Have you encountered that with the increased number of GPUs, the speed will get significantly slow?  I follow your template for training CC12M dataset and when I use 8 GPUs, the speed is 2.02iter/s. However, when I use 32 GPUs,  i.e., 4 machines, the speed is only 0.22iter/s.

Hi, Thanks for your code. Have you encountered that with the increased number of GPUs, the speed will get significantly slow? I follow your template for training CC12M dataset and when I use 8 GPUs, the speed is 2.02iter/s. However, when I use 32 GPUs, i.e., 4 machines, the speed is only 0.22iter/s.

yes, it's possible in multi-node (multiple machine, each have multiple GPUs) training actually ~
it's a common issue we may encountered in multi-node environment ~
The first sad truth reveal in the following story :
"Suppose i have 8 gpu in one node 0.6-iter/s, now i have 2 node (16 gpus), could i expect the 2xspeed (1.2-iter/s) ?
In theory, yes. However, it have several factors largely decrease the training speed :

Network bandwidth between each node. If the sending data is in large size (see multi-node training, it'll sync the gradient for each network weights, ...etc.), it'll occupy the network bandwidth and largely decreasing the speed.
Each gpu is stay on a process, it may trigger 8 process per node (suppose one node have 8 gpus). Then the communication overhead between the processes will also decrease the speed.
so, it's completely possible slower then the previous one node setup.
"
I have the experience applied 2 node training on azure, since i setup the outter network routing (didn't go through by the virtual LAN) it even slower then one node.

If you're hosting 4 machines, it's lucky you can do something for the above factors i mentioned. If you can not do that, then my last suggestion is that you can use 4 machine to run the different experimental setup if you have.
Good luck, happy coding ~

Hi! If I using the .with_epoch() function for mulit-node training, should I also divide the get_world_size() inside .with_epoch()?

Hi! If I using the .with_epoch() function for mulit-node training, should I also divide the get_world_size() inside .with_epoch()?
The upper-bound iteration in the progress bar never changed, no matter how many GPU I set up in the pl ddp trainer.

Unfortunately, i had tried several methods in document (with_epoch is included), but still cannot get the correct results even in single node (so i had gave up this way and create this issue with my solution).
So, i can't extend those method in multi-node environment and provide the correct information. For this issue, you may ask the main contributor @tmbdev for help (he knows more in this case).

@HuangChiEn Thanks for your reply! When I set resampled=True, the speed is relatively normal. I think that parameter will benefit. However, I have another question is it possible to do shardshuffle=True? Since the code reveals that each shard can be viewed as a webdataset, it seems that the operation shuffling between shards is hard.

resampled=True

sorry to reply late ~
i think resample=True can speedup by fetching the batched data from their dataloader on process. Namely, it will not fetch the identical dataloader, lead the progressbar show the longer step number.
For example.
Suppose we have 32 samples in dataloader and setup the batch size 4. If i have single gpu, it take 8=(32/4) steps.
However, if i run on two gpu with the same batch size, it will still run the 8 steps (because each gpu fetch from different dataloader)
// just my experience for sharing..
First, it depend on the property of dataset, not all dataset need to be shuffled.
(1) The dataset need to shuffle : (i.e. sorted classification dataset)
dataset -> { [0~100, dog], [101~200, cat], ... so on }, in this case we if doesn't shuffle the dataset, model may be overfit to the pattern that first learn all dog features, then learn all cat features, and so on.
(2) The dataset may not need to shuffle : (i.e. image-text pair dataset)
Laion 115M crawl from website, it doesn't have such pattern like above. Each image-text pair is independent from its order in dataset.

In my case (Laion 115M), i don't strictly need to shuffle. But i first wrap webdataset as a single stream iterator, then the default sampler for torch multi-gpu DistributedSampler, it automatically shuffle all data samples indices, then take the subset for each gpu (it equivalent to shard shuffle).
The way you mentioned is also available ~ (shuffle urls or shuffle indices)

@HuangChiEn  Thanks for your reply! I think in your code, we can not set resample=True, since the for loop of the dataset.
for batch_id, sample in enumerate(dataset):

However, without setting resampled=True, each time the work will try to read data from the disk, making it relatively slow. I wonder if there exists some efficient way to solve the problem by setting resampled=True and automatically breaking the for loop simultaneously.

@HuangChiEn Thanks for your reply! I think in your code, we can not set resample=True, since the for loop of the dataset.
for batch_id, sample in enumerate(dataset):

However, without setting resampled=True, each time the work will try to read data from the disk, making it relatively slow. I wonder if there exists some efficient ways to solve the problem by setting resampled=True and automatically breaking the for loop simultaneously.

Umm.. let's say the for-loop simulate the general step in training loop. In torchlightning, train_step will fetch the batch data from for-loop underhood. So, i'm don't think for batch_id, sample in enumerate(dataset) become the issue.
On the other hand, data sample is placed in disk basically. So, torch provide some flag, such as pin_memory, prefetch_factor to allow us load the samples in memory before feed it into GPU.

see all flags in https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader
If the slow speed cause by reading data from the disk, you can try to setup those flags.

Lastly, if you want to setup resampled=True in this scenario, you may open a new issue to text the author of webdataset.

@HuangChiEn Thank you for your code！！！ But i get the following error, after the first step, do you have any idea?

Could you list your package version ?
pip list (wanna chk webdataset, torch, torchlightning)
As the error message, it raised from webdataset :

Is it possible to have a minimum reproduction :
# single worker
dataset = wds.WebDataset(url).shuffle(1000).decode('pilrgb', handler=wds.warn_and_continue).to_tuple("jpg", "txt", handler=wds.warn_and_continue).map(self.trfs)
for batch_id, sample in enumerate(dataset):
    pass

If it pass, then i guess it's caused by multiple workers.

thank you for reply !!!! @HuangChiEn
torch                     2.4.1+cu124
webdataset                0.2.100
lightning                 2.4.0
when i change my code to

it pass,

thank you for reply !!!! @HuangChiEn torch 2.4.1+cu124 webdataset 0.2.100 lightning 2.4.0
when i change my code to
it pass,
and when i try to Alignment the version of webdatset to 0.2.86 the bug disappear, but i dont no why the next url is printed

Hello, glad to hear that the bug disappear. You can comment out the print(urlll...) to filter out the message. While this print function place in there without the rank condition (if rank==0: print(...)), so all of the rank will print this information.
As your screen shot, each process actually will first received each url, but the code snippet below will filter out the url not belong to the rank :
def __iter__(self):
        process_rank = get_rank()
        world_size = get_world_size()
        for url in self.urls:
            # all url will be received by each process!!
            dataset = wds.WebDataset(url, nodesplitter=wds.split_by_worker).shuffle(1000).decode('pilrgb', handler=wds.warn_and_continue).to_tuple("jpg", "txt", handler=wds.warn_and_continue).map(self.trfs)
            for batch_id, sample in enumerate(dataset):
                if batch_id % world_size == process_rank:
                    yield sample

                # drop out the batch sample which not belong to this process,
                # respect with the process rank!!
                else:
                    continue


RevEng E2E testing

Move common E2E testing functionality into a common project.
Adds e2e testing for reverse engineering for SQLite.
Fix #2721

cc: @lajones

:shipit:


Question on running a session twice

Hi,
after some time of using the property of the tf_manager to run a session number of times, I realized it might be a wrong approach for me. But not sure, so I ask for second opinion. The code is here: https://github.com/StoyanVenDimitrov/neuralmonkey/blob/with_new_decoder/neuralmonkey/trainers/mrt_trainer.py
The case is:
I use the decoder.py in sampling mode. It gives me some logits and vocabulary IDs. I run a session with fetches only for the IDs and obtain the sentences they represent and their GLEU score. Then I go back to gather the logits that these IDs stand for, running another session with fetches for the loss, where these gathered logits are part of. But this should be wrong. Because even if I have no fetches that involve logits on the first run, on the second run, because of the loss fetch, new logits are obtained in the decoder, right? I thought they aren't because no fetches for them have been ran yet. But session.run() should result in recomputing all nodes that are involved in the fetches. This means session.partial_run() is the only way. Am I right?   Thanks a lot!
Best,
Stoyan

Hi, sorry for not replying for a long time - did you solve the problem?
2017-09-19 16:08 GMT+02:00 StoyanVenDimitrov<EMAIL_ADDRESS>
Closed #531 https://github.com/ufal/neuralmonkey/issues/531.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ufal/neuralmonkey/issues/531#event-1255521720, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABwcs1n9dcAOaQU_Xx8sUhhI-67M5kbYks5sj8rpgaJpZM4PSC44
.

Hi,
No worries, it was rather a general TF question.
Yes, I just ran the second session not in sample, but in train modus, using the output of the sampling as input for the second session.


How does an organization build a team to set up and run a governance program.

Using Coco Pharmaceuticals persona to drive the scenario, provide Egeria capability with supporting resources to explain how Jules Keeper, the new CDO, builds a team to beef up their governance programs.
This scenario needs to show how Egeria helps teams focused on different governance domains to collaborate to create an integrated governance ecosystem for the business.

This scenario is implemented solely by the governance program OMAS.  It uses the concept of a governance officer (from area 4) and links it to a person's profile (from area 1) to show the appointment of the individual to the governance officer post.  The link is implemented by the GovernancePost relationship (area 4).  In this scenario, it exercises the effectivity dating capablity supported by OMRS to allow predecessors and successors to be defined for the post.
An important development that came from this work is the use of the GovernanceDomain enum.  The definitions of the governance officer and some of the governance definitions were generalized to support governance from any governance domain, not just data governance.  Thus in the sample, you see Jules Keeper, the Coco Pharmaceuticals CDO organizing the organizations leaders to identifier governance officers for each of the domains that need attention.
See: the description of the sample scenario and  the sample code.  The sample code is callng the Governance Leadership Interface of the Governance Program OMAS


SSL_write / tls_write buffer pointer is not read only

My code is strict about whatever target memory may or may not be modified, so I use const keyword extensively where it matters. However SSL_write's buf pointer is just void *, not const void *. The same is with tls_write. Compiler gives me a warning about that because my wrapper function accepts const void *.
Even my OpenSSL 1.0.0l manual page says buf pointer is const void *.
The fix should be trivial.

Can you confirm it is ok now?

Yes it is all ok now. Thank you for your hard work providing such a good project.


Picture Frames

What version of ViaVersion are you using? Type /ver viaversion
0.7.3
What version of Spigot are you using? Type /ver
spigot 1.8.8 latest
What plugins are you using? Type /plugins
Plugins (6): PluginManagerReloaded, WorldEdit, ViaVersion, PacketListenerApi, MapManager, AnimatedFrames
Is there an error in the console? Use pastebin.com. Is there a kick message?
http://hastebin.com/xusaziluwe.vhdl

Can you turn debug=true in server.properties with viaversion debug by doing /viaversion debug also use the entire log.

http://pastebin.com/S0HLHbLB

Could you please provide us more information on what the exact issue is? Or how to replicate? @mat41997

java 8
client 1.9
command:
/af create http://a.disquscdn.com/uploads/mediaembed/images/3445/2254/original.jpg?w=600&h
files
http://speedy.sh/zezJq/ZSer.7z
or
https://drive.google.com/file/d/0B4sNCwuKX5m8LWc2eTdPWk5MUXc/view?usp=sharing
skan
https://www.virustotal.com/pl/file/13966195351c0d188aff52a13f51f835daf83eae9870852943f587dd385459fd/analysis/1459676446/

Thanks for providing all this information, I can indead confirm this crash @mat41997 ;)
I'll look into in it.

I've looked into this and after decompiling MapManager the issue is it sends invalid data, we could patch it but really the plugin shouldn't be doing this I hope to be in contact with inventivtalent to get this fixed.

I've contacted inventive and got him to fix the issue.
https://www.spigotmc.org/resources/api-mapmanager-1-7-1-8-1-9.19198/updates
New one should work :)


Release artifact version string differs from what actual binary returns

SysOps minor issue related to consistency and conventions.
I'm building small automation script to setup cardano-wallet component on a node - why not :)
So for example: Release tag in the git repo and github releases web UI says v2021-05-26
But actual binaries return
cardano-wallet version
2021.5.26 (git revision: 7426ccc17ecffcc112abf5e8382bcb89cb24f771)

Difference in v, -, 0
while it is easy to manipulate outputs, like bellow, imho better consistency is always beneficial.
# automation done using ansible
- name: Normalizing cardano release version part1
  set_fact:
    normalized_wallet_new_v_list: "{{ wallet_v.split('-') }}"

- name: Normalizing cardano release version part2
  set_fact:
    normalized_wallet_new_v: "{{ normalized_wallet_new_v_list[0] | replace('v', '') }}.{{ normalized_wallet_new_v_list[1] | replace('0', '') }}.{{ normalized_wallet_new_v_list[2] | replace('0', '') }}"

# actual binary version is caught like so and it is ok to skip right-most part of the output:
cardano-wallet version | grep -Eo "[0-9]+\.[0-9]+\.[0-9]+"

Above is still error-prone because that snippet removes zero from such digits like 10, 20, 30 too. Of course, there are dozen bash methods to achieve the required result. - But hopefully, you got the idea, what it takes to compare not-consistent version strings.

Yes OK, thanks for reporting.
It's showing the Cabal package version and the Cabal package manager disallows leading zeroes in version numbers.
But for the git tags (shown in GitHub releases page), we like to use the nicer date formatting.
Perhaps in the cardano-wallet version CLI we could format the Cabal version to match the git tag.


can i disable async ?

I assume you mean not use async/await? Your can you regular promises (.then) calls if that's what you prefer.

@ebidel can you give me example from screenshot code example ?

const puppeteer = require('puppeteer');
puppeteer.launch().then(browser => {
  browser.newPage()
      .then(page => page.goto('https://example.com'))
      .then(resp => page.screenshot({path: 'example.png'}))
      .then(buffer => browser.close());
});

https://developers.google.com/web/fundamentals/getting-started/primers/async-functions is a good resource.

Awesome, should add this to the code examples. Aws lambda just released nodejs 6.10

It`s really useful for me

@ebidel code doesn't seem working with node v10.4.1 + puppeteer v1.5.0:
(node:19155) UnhandledPromiseRejectionWarning: ReferenceError: page is not defined
at browser.newPage.then.then.resp (~/program/puppeteer/test.js:5:26)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:19155) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:19155) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Here is a working version of the code above (fixed UnhandledPromiseRejectionWarning):
const puppeteer = require('puppeteer');

puppeteer.launch().then(browser => {
  browser.newPage()
    .then(page => {
      page.goto('https://www.google.com')
        .then(resp => page.screenshot({path: 'example.png'}))
        .then(buffer => browser.close());
    });
});

This isn't a clean option for some use cases. I'm trying to migrate from Selenium (Java) to Puppeteer so that I can take advantage of headless Chrome in GC Functions. However, Selenium code in Java is way easier to maintain if you require synchronicity.  If I were to convert my code to synchronous Puppeteer, there'd be nested promises trailing off to the right.

@karthigb we strongly recommend using async/await when working with Puppeteer; this should make migration much easier. The only case where you can't use async/await is running in Node6, but Node6 will be end-of-lifed in half a year, so I'd suggest to not use it for anything new.

@aslushnikov Thank you :)
I have a Selenium script that automates web application form processing. I have it stepping through a twenty page application process that must be done sequentially.
How can I accomplish this cleanly with async/await?

Asynchronous presents a problem for me.
I have 10-15 questions that need to be answered across numerous pages. The questions must be answered in the right order. The number of questions answered, and the question pagination will depend on the answers provided.
Selenium for Java works. Selenium for Node does not.
I would love to move on to GCF, but fear this is a blocker.

Selenium for Java works. Selenium for Node does not.

@karthigb Puppeteer has no relation to Selenium; Puppeteer is a separate product with a separate codebase.

I have 10-15 questions that need to be answered across numerous pages. The questions must be answered in the right order. The number of questions answered, and the question pagination will depend on the answers provided.

This is a typical scenario for puppeteer; I'd encourage you to give it a try.

@aslushnikov  Puppeteer has no relation to Selenium? Crap, seriously!?!?!

@aslushnikov I'll give it another go ;)
But I'm also running into this:
https://github.com/GoogleChrome/puppeteer/issues/3223

@aslushnikov I owe you a big thanks. Everything works beautifully. Puppeteer is much faster and more reliable than Selenium. Being able to run it from GCF will save me a lot of overhead. Thank you!

Here is a working version of the code above (fixed UnhandledPromiseRejectionWarning):
const puppeteer = require('puppeteer');

puppeteer.launch().then(browser => {
  browser.newPage()
    .then(page => {
      page.goto('https://www.google.com')
        .then(resp => page.screenshot({path: 'example.png'}))
        .then(buffer => browser.close());
    });
});

how can I fill input in search-box of the website in this code ? giving me few errors

Here is a working version of the code above (fixed UnhandledPromiseRejectionWarning):
const puppeteer = require('puppeteer');

puppeteer.launch().then(browser => {
  browser.newPage()
    .then(page => {
      page.goto('https://www.google.com')
        .then(resp => page.screenshot({path: 'example.png'}))
        .then(buffer => browser.close());
    });
});

Hello can i have on help please here, how can I fill input in search-box of the website in this code ? giving me few errors... like
throw new Error('Execution context was destroyed, most likely because of a navigation.');
^
Error: Execution context was destroyed, most likely because of a navigation.


gtk_layer_set_exclusive_zone

A Very Humble Request: could we also have gtk_layer_set_exclusive_zone implemented? :innocent:
[edit] Sorry, nothing urgent, as it doesn't resolve my current problem, as it turns out.

didnt see the edit in time sooo: https://github.com/dlasky/gotk3-layershell/commit/5cca0b8192612d5ed627efc41f497a49ca84333c

what problem are you running into?

Nothing serious. I want my dock to be able to mimic various docks. When started as in Ubuntu, it overlaps the panel. If panel has been started after the dock, it looks all right:

If I start the dock after the panel, it behaves oddly:

Actually I'm not going to use the dock like this - if at all.

For daily use it looks better like this:

And many thanks for adding what I asked for!

No problem! Yeah unfortunately there are only a few layer choices otherwise I would say to bump the bar to a higher index but then it would likely overlap the Lock Screen as well.

I wish I could run sway on my desktop as well.  Did they change this?

it launches with --my-next-gpu-wont-be-nvidia but then running any other app and the CPU seems to spike to 800% (all cores maxed)

Shit. I bought my current all-AMD laptop with sway in mind, but it also would give me a big headache initially, until kernel 5.8.


Consider supporting default AWS profiles

It would cause less friction if default AWS profiles could be used in addition to named profiles.

There's no reason we don't let user run with default AWS profile.
setup command was just a draft, ECS context creation UX need to be fully revisited

cc @CarmenAPuccio


feat/twemproxyconfig-e2e-tests

This PR adds e2e tests for the TwemproxyConfig custom resource.
I also moved all the code related to twemproxy types and configs to the same package, which seemed like a better way of organizing it.
/kind feature
/priority important-soon
/assign

/retitle WIP feat/twemproxyconfig-e2e-tests
seems like I have a flaky test ...

/retitle feat/twemproxyconfig-e2e-tests

@raelga this can now be reviewed

/approve


Set offset to 0 when filtering emojis

Details
When we filter emojis, we should scroll back to the top of the list
Fixed Issues

$ https://github.com/Expensify/App/issues/15347
PROPOSAL: N/A
Tests
Web/Desktop only, did not modify anything mobile related

Open any chat
Open the Emoji Picker
Scroll down at least 1 full page
Search for the letter "a"
Make sure you get brought back to the top instead of staying scrolled

[x] Verify that no errors appear in the JS console

Offline tests
N/A
QA Steps
Web/Desktop only, did not modify anything mobile related

Open any chat
Open the Emoji Picker
Scroll down at least 1 full page
Search for the letter "a"
Make sure you get brought back to the top instead of staying scrolled

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
https://user-images.githubusercontent.com/42391420/221308603-82fea2af-b74d-4e37-acc3-45602fa31ec1.mp4

Mobile Web - Chrome
N/A

Mobile Web - Safari
N/A

Desktop
https://user-images.githubusercontent.com/42391420/221308644-ad3d5f2d-3bbf-4c76-853c-a6972878a249.mp4

iOS
N/A

Android
N/A

@thesahindia @neil-marcellini One of you needs to copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

🚀 Deployed to staging by https://github.com/neil-marcellini in version: 1.2.77-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
failure ❌

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to staging by https://github.com/neil-marcellini in version: 1.2.77-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/yuwenmemon in version: 1.2.77-4 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Fix mock kind of Order builder

Summary
The kind of an Order can be any form of OrderKind. Currently the builder for the entity only returns OrderKind.Buy. This changes it to also return OrderKind.Sell.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 49.274%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
4068

Relevant Lines:
6658

💛  - Coveralls


Plug CADL ranch into CI

This should include CI validation and CADL ranch report:
cc @timotheeguerin for help if neeeded

@haolingdong-msft
See a cadl-ranch-coverage.json generated. Please follow up with cadl-range on reporting (if available).

@haolingdong-msft
See a cadl-ranch-coverage.json generated. Please follow up with cadl-range on reporting (if available).

sure

I did add support for coverage collection which will create this cadl-ranch-coverage.json. By itself this will not complain about anything. You can use cadl-ranch check-coverage command to validate that this coverage file are covering 100% of the cases.
https://github.com/Azure/cadl-ranch/blob/56fa076c030275b69952094515bb5f8eb4984136/docs/using-cadl-ranch.md#validate-and-merge-coverage
For coverage reporting I haven't gotten to it just yet but hopefully will have that soon.

@haolingdong-msft
For now, you can just print the result of "cadl-ranch check-coverage" in CI.
Though I am not sure whether it requires stopping the server.

current 81%
will do the rest in MQ https://github.com/Azure/autorest.java/issues/1863


NoReverseMatch: Reverse for 'account_confirm_email' with arguments

I have created an app named 'api' within a project and the api/urls.py looks like:
...
url(r'^accounts/', include('allauth.urls')),
url(r'^rest-auth/registration/account-confirm-email/(?P<key>\w+)/$',confirm_email, name='account_confirm_email'),
url(r'^rest-auth/', include('rest_auth.urls')),
url(r'^rest-auth/registration/', include('rest_auth.registration.urls')),
...

but when trying to register I get the following stacktrace:
[2016-02-15 23:45:35,093] ERROR [base:handle_uncaught_exception:256 8977] Internal Server Error: /api/rest-auth/registration/
Traceback (most recent call last):
  File "/home/.../webapps/project_test/lib/python2.7/Django-1.8.7-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/.../webapps/project_test/lib/python2.7/Django-1.8.7-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/.../webapps/project_test/lib/python2.7/Django-1.8.7-py2.7.egg/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/.../webapps/project_test/.ve/lib/python2.7/site-packages/rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "/home/.../webapps/project_test/.ve/lib/python2.7/site-packages/rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/.../webapps/project_test/.ve/lib/python2.7/site-packages/rest_auth/registration/views.py", line 44, in post
    self.form_valid(self.form)
  File "/home/.../webapps/project_test/.ve/lib/python2.7/site-packages/rest_auth/registration/views.py", line 36, in form_valid
    self.get_success_url())
  File "/home/.../webapps/project_test/.ve/lib/python2.7/site-packages/allauth/account/utils.py", line 157, in complete_signup
    signal_kwargs=signal_kwargs)
  File "/home/.../webapps/project_test/.ve/lib/python2.7/site-packages/allauth/account/utils.py", line 114, in perform_login
    send_email_confirmation(request, user, signup=signup)
  File "/home/.../webapps/project_test/.ve/lib/python2.7/site-packages/allauth/account/utils.py", line 286, in send_email_confirmation
    signup=signup)
  File "/home/.../webapps/project_test/.ve/lib/python2.7/site-packages/allauth/account/models.py", line 60, in send_confirmation
    confirmation.send(request, signup=signup)
  File "/home/.../webapps/project_test/.ve/lib/python2.7/site-packages/allauth/account/models.py", line 121, in send
    activate_url = reverse("account_confirm_email", args=[self.key])
  File "/home/.../webapps/project_test/lib/python2.7/Django-1.8.7-py2.7.egg/django/core/urlresolvers.py", line 578, in reverse
    return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs)))
  File "/home/.../webapps/project_test/lib/python2.7/Django-1.8.7-py2.7.egg/django/core/urlresolvers.py", line 495, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns))
NoReverseMatch: Reverse for 'account_confirm_email' with arguments '(u'vgro7mdagterfbnqxlkihpnzs2bsp1iktwxng9vecznpeg8hcdv1eidco11tee3u',)' and keyword arguments '{}' not found. 0 pattern(s) tried: []

Using a shell I see:
python manage.py shell
Inside api.apps.ApiConfig.ready
Python 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.core.urlresolvers import reverse
>>>
>>> reverse('account_confirm_email', args=['l8ig7wqo1l4ypd7uosuoxlgnv2qxf5voe5ehwcd7s46hbprzyx9qwjgeontw5un6'])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/divkis01/webapps/instaguide_test/lib/python2.7/Django-1.8.7-py2.7.egg/django/core/urlresolvers.py", line 578, in reverse
    return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs)))
  File "/home/divkis01/webapps/instaguide_test/lib/python2.7/Django-1.8.7-py2.7.egg/django/core/urlresolvers.py", line 495, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns))
NoReverseMatch: Reverse for 'account_confirm_email' with arguments '('l8ig7wqo1l4ypd7uosuoxlgnv2qxf5voe5ehwcd7s46hbprzyx9qwjgeontw5un6',)' and keyword arguments '{}' not found. 0 pattern(s) tried: []
>>>

while if I use api:account_confirm_email, it does return correct url.
>>> reverse('api:account_confirm_email', args=['l8ig7wqo1l4ypd7uosuoxlgnv2qxf5voe5ehwcd7s46hbprzyx9qwjgeontw5un6'])
u'/api/rest-auth/registration/account-confirm-email/l8ig7wqo1l4ypd7uosuoxlgnv2qxf5voe5ehwcd7s46hbprzyx9qwjgeontw5un6/'

Given the exception is from within the allauth and I guess modifying the allauth is not the right solution here, what am I missing here or if it is an issue in rest-auth itself?

When you include urls.py from your main module, you specify namespace="api", try to delete that statement or directly include "account_confirm_email" url to your main module.
Really that trouble not about rest-auth or allauth, that is about how django importing and resolve urls patterns.

@Akay7 What if I wish to provide the urls provided by rest-auth and all-auth as sub path to api? For example instead of /rest-auth/registration/ it should be available as /api/rest-auth/registration/, in that case   what would be the effect of deleting the namespace?

@divkis01 api:account_confirm_email - in that part "api" is name-space you must delete it, and after that your statement:
>>> reverse('account_confirm_email', args=['l8ig7wqo1l4ypd7uosuoxlgnv2qxf5voe5ehwcd7s46hbprzyx9qwjgeontw5un6']) will work, and i hope your application will work too.

@Akay7 Yup this seems to work. It seems to make sense now. Thanks for your help. Closing the ticket as this is not really an issue with rest-auth or allauth.


User profile page

The MySkills WebApp should provide a specific page where the authenticated user can view her/his user profile data and modify profile-specific settings in the future.
The should be a separate navigation link to open the user profile page, e.g. by clicking on the user name in the title bar.
The user profile page should only display a read-only view of the "userName", "firstName", "lastName" and "email" properties of the user because these values are obtained from the OAuth2 identity token.

Closing. Feature has been implemented. The layout can be improved but this is subject to a future optimization.


Currency exchange conversion

Allow specifying invoice amounts in other currencies and convert to bitcoins. The invoice should lock-in the exchange rate for some pre-determined time, then expire.

Invoice expiry was opened as a separate issue (#6).


build fails with rustlang/rust:nightly docker container

I'm having trouble compiling lettre_email on the rustlang/rust:nightly docker image. I'm not doing anything special, lettre is included as 0.9 in Cargo.toml. However during build, rustc complains about being unable to resolve some type without an annotation:
   Compiling lettre_email v0.9.1
error[E0283]: type annotations required: cannot resolve `std::string::String: std::convert::AsRef<_>`
   --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/lettre_email-0.9.1/src/lib.rs:139:66
    |
139 |         self.header(("Content-Type", format!("{}", content_type).as_ref()))
    |                                                                  ^^^^^^

For the full log, see https://gitlab.com/sequoia-pgp/hagrid/-/jobs/220897029
Environment (please complete the following information):

Lettre version: 0.9.1
OS: docker container based on debian stretch (rustlang/rust:nightly)

This is failing in beta as well.

Fixed in 0.9.2.


While a new flavor release is building, the NaaVRE base image list is empty

When $BASE_IMAGE_TAGS_URL is not defined, we fall back to https://github.com/QCDIS/NaaVRE-flavors/releases/latest/download/base_image_tags.json in jupyterlab_vre/component_containerizer/handlers.py#L263.
However, when creating a new release on NaaVRE-flavors, there is a ~45-minutes window during which https://github.com/QCDIS/NaaVRE-flavors/releases/latest/download/base_image_tags.json points to an empty document. When creating a release, we do:

Create the release (changes where https://github.com/QCDIS/NaaVRE-flavors/releases/latest/ points to)
Build all images (~45 minutes)
Generate base_image_tags.json and add it to the release

Possible solutions:

always set BASE_IMAGE_TAGS_URL in deployments
ship a version of https://github.com/QCDIS/NaaVRE-flavors/releases/latest/download/base_image_tags.json in the naavre-jupyter docker images and use this as fallback

We always set BASE_IMAGE_TAGS_URL in deployments to point it to a specific version eg. https://github.com/QCDIS/NaaVRE-flavors/releases/download/v0.14/base_image_tags.json


Switch back to snapshot images in samples

Fixes #195

we discussed this yesterday and decided to leave samples on the latest stable version (devs can always change locally)... however, we can reopen the debate if you'd like. The argument in favor of changing to snapshots is that we do have a v0.0.1 tagged version of the riff repo: https://github.com/projectriff/riff/tree/v0.0.1
cc @trisberg

I'd like to keep the samples at 0.0.1 for now, makes it easier to run them locally and on GKE. If I do local dev work I can change the sample. Just my 2c.

closing for now


Oracle to Spanner Queries Report

As a CE and/or SCE, I want to gather ALL relevant database information in a single report to modernize the database to Spanner. So, I can manually analyze what would be the challenges.
Expected Results:

Collection of Queries to create a Oracle to Spanner Report
Create a Dashboard or HTML Report or Any other Report format

Initial Target Completion:
21 / Q4

with v_dbobjects as
(select pkey,
sum(tables) tables,
sum(tablespartitioned) tablespartitioned,
sum(tablessubpartitioned) tablessubpartitioned,
sum(indexes) indexes,
sum(lobs) lobs,
sum(synonyms) synonyms,
sum(views) views,
sum(mviews) mviews,
sum(triggers) triggers,
sum(sequences) sequences,
sum(functions) functions,
sum(procedures) procedures,
sum(packages) packages,
sum(packagesbody) packagesbody,
sum(dblinks) dblinks
from
(
select pkey,
CASE
WHEN TRIM(objecttype) = 'TABLE'
THEN cast(coun as numeric)
END tables,
CASE
WHEN TRIM(objecttype) = 'TABLE PARTITION'
THEN cast(coun as numeric)
END tablespartitioned,
CASE
WHEN TRIM(objecttype) = 'TABLE SUBPARTITION'
THEN cast(coun as numeric)
END tablessubpartitioned,
CASE
WHEN TRIM(objecttype) = 'INDEX PARTITION'
THEN cast(coun as numeric)
END indexespartitioned,
CASE
WHEN TRIM(objecttype) = 'INDEX'
THEN cast(coun as numeric)
END indexes,
CASE
WHEN TRIM(objecttype) = 'LOB'
THEN cast(coun as numeric)
END lobs,
CASE
WHEN TRIM(objecttype) = 'SYNONYM'
THEN cast(coun as numeric)
END synonyms,
CASE
WHEN TRIM(objecttype) = 'VIEW'
THEN cast(coun as numeric)
END views,
CASE
WHEN TRIM(objecttype) = 'MATERIALIZED VIEW'
THEN cast(coun as numeric)
END mviews,
CASE
WHEN TRIM(objecttype) = 'TRIGGER'
THEN cast(coun as numeric)
END triggers,
CASE
WHEN TRIM(objecttype) = 'SEQUENCE'
THEN cast(coun as numeric)
END sequences,
CASE
WHEN TRIM(objecttype) = 'FUNCTION'
THEN cast(coun as numeric)
END functions,
CASE
WHEN TRIM(objecttype) = 'PROCEDURE'
THEN cast(coun as numeric)
END procedures,
CASE
WHEN TRIM(objecttype) = 'PACKAGE'
THEN cast(coun as numeric)
END packages,
CASE
WHEN TRIM(objecttype) = 'PACKAGE BODY'
THEN cast(coun as numeric)
END packagesbody,
CASE
WHEN TRIM(objecttype) = 'DATABASE LINK'
THEN cast(coun as numeric)
END dblinks
from oracle-playground.Walmart.dbobjects a
where trim(in_owner) = '0' and trim(in_object_type) = '0' and trim(in_con_id) = '0' and trim(in_editionable) = '0'
)
group by pkey),
v_dbobjectssize as
(select pkey, sum(tables_gb) tables_gb, sum(tablespartitioned_gb) tablespartitioned_gb,
sum(tablessubpartitioned_gb) tablessubpartitioned_gb, sum(indexes_gb) indexes_gb,
sum(indexespartitioned_gb) indexespartitioned_gb, sum(lobs_gb) lobs_gb
from (
select pkey,
CASE
WHEN TRIM(segment_type) = 'TABLE'
THEN cast(total_gb as numeric)
END tables_gb,
CASE
WHEN TRIM(segment_type) = 'TABLE PARTITION'
THEN cast(total_gb as numeric)
END tablespartitioned_gb,
CASE
WHEN TRIM(segment_type) = 'TABLE SUBPARTITION'
THEN cast(total_gb as numeric)
END tablessubpartitioned_gb,
CASE
WHEN TRIM(segment_type) = 'INDEX PARTITION'
THEN cast(total_gb as numeric)
END indexespartitioned_gb,
CASE
WHEN TRIM(segment_type) = 'INDEX'
THEN cast(total_gb as numeric)
END indexes_gb,
CASE
WHEN TRIM(segment_type) = 'LOB'
THEN cast(total_gb as numeric)
END lobs_gb
from oracle-playground.Walmart.spacebyownersegtype
)
group by pkey),
v_datatypes as
(
select pkey,
sum(dt_number) dt_number,
sum(dt_varchar2) dt_varchar2,
sum(dt_timestamp) dt_timestamp,
sum(dt_raw) dt_raw,
sum(dt_date) dt_date,
sum(dt_char) dt_char,
sum(dt_clob) dt_clob,
sum(dt_blob) dt_blob,
sum(dt_float) dt_float,
sum(dt_xmltype) dt_xmltype
from (
select pkey,
CASE
WHEN TRIM(data_type) = 'NUMBER'
THEN cnt
END dt_number,
CASE
WHEN TRIM(data_type) = 'VARCHAR2'
THEN cnt
END dt_varchar2,
CASE
WHEN substr(TRIM(data_type),0,9) = 'TIMESTAMP'
THEN cnt
END dt_timestamp,
CASE
WHEN TRIM(data_type) = 'RAW'
THEN cnt
END dt_raw,
CASE
WHEN TRIM(data_type) = 'DATE'
THEN cnt
END dt_date,
CASE
WHEN TRIM(data_type) = 'CHAR'
THEN cnt
END dt_char,
CASE
WHEN TRIM(data_type) = 'CLOB'
THEN cnt
END dt_clob,
CASE
WHEN TRIM(data_type) = 'BLOB'
THEN cnt
END dt_blob,
CASE
WHEN TRIM(data_type) = 'FLOAT'
THEN cnt
END dt_float,
CASE
WHEN TRIM(data_type) = 'LONG'
THEN cnt
END dt_long,
CASE
WHEN TRIM(data_type) = 'XMLTYPE'
THEN cnt
END dt_xmltype
from (
select pkey, data_type, sum(cast(coun as numeric)) cnt
from oracle-playground.Walmart.datatypes
group by pkey, data_type
)
)
group by pkey
),
v_sourcecode as
(
select pkey,
sum(cast(nr_lines as numeric)) nr_lines,
sum(cast(qt_objs as numeric)) qt_objs,
sum(cast(nr_lines_w_utl as numeric)) nr_lines_w_utl,
sum(cast(nr_lines_w_dbms as numeric)) nr_lines_w_dbms,
sum(cast(nr_lines_w_exec_im as numeric)) nr_lines_w_exec_im,
sum(cast(nr_lines_w_dbms_sql as numeric)) nr_lines_w_dbms_sql,
sum(cast(nr_lines_w_dbms_utl as numeric)) nr_lines_w_dbms_utl,
sum(cast(nr_lines_total as numeric)) nr_lines_total
from oracle-playground.Walmart.sourcecode
group by pkey
)
select a.pkey, d.dbversion, d.redo_gb_per_day, d.rac_dbinstaces, d.db_size_allocated_gb, d.db_size_in_use_gb,
a.tables, b.tables_gb, a.tablespartitioned, b.tablespartitioned_gb, a.tablessubpartitioned, b.tablessubpartitioned_gb,
a.indexes, b.indexes_gb, b.indexespartitioned_gb, a.lobs, b.lobs_gb, a.synonyms, a.views, a.mviews, a.triggers, a.sequences,
a.functions, a.procedures, a.packages, a.packagesbody, a.dblinks,
c.dt_number, c.dt_varchar2, c.dt_timestamp, c.dt_raw, c.dt_date, c.dt_char, c.dt_clob, c.dt_blob, c.dt_float, c.dt_xmltype,
e.nr_lines nr_lines_code, e.qt_objs nr_plsql_objs, e.nr_lines_w_dbms nr_lines_code_w_dbms, e.nr_lines_w_exec_im nr_lines_code_w_exec_im,
e.nr_lines_w_dbms_sql nr_lines_code_w_dbms_sql, e.nr_lines_w_dbms_utl nr_lines_code_w_dbms_utl
from v_dbobjects a
inner join v_dbobjectssize b
on trim(a.pkey) = trim(b.pkey)
inner join v_datatypes c
on trim(a.pkey) = trim(c.pkey)
left outer join oracle-playground.Walmart.dbsummary d
on trim(a.pkey) = trim(d.pkey)
left outer join v_sourcecode e
on trim(a.pkey) = trim(e.pkey);


App: Implement HTTP/HTTPS support for logging server

Currently system relies on a basic, unencrypted TCP/UDP packet for passing data to a remote logging server. As our client may input sensitive information (ex. Medical Info) in areas such as ReasonRequired, best to add some extra protection especially if a malicious actor may be sniffing network traffic on the client site.
Responsible code for current remote logging system

https://github.com/ripeda/macOS-enterprise-privileges/blob/5ab148e55a4aa2ca44974e3c6c64d4dba72e192e//source/PrivilegesHelper/PrivilegesHelper.m#L263-L266

Switching to HTTP/HTTPS also allows for greater stability since we can rely on libraries such as FastAPI which has been proven to be extremely resilient. Should reduce the amount of logic required in our Privileges-Logging-Server.

We'll likely want to also implement a public and private key system, where clients will encrypt the packet using a public key that only the server's private key can decrypt.

Project now supports both HTTP and HTTPS, however testing HTTPS will need extra work. Currently our self-signed certificate is not valid:
SAPCorp: ERROR! Remote logging failed: Error Domain=NSURLErrorDomain Code=-1202 "The certificate for this server is invalid. You might be connecting to a server that is pretending to be “<IP_ADDRESS>” which could put your confidential information at risk."


Fixes error when calling upgradeToRevocableSession without a sessionToken

Fixes: #2720
Matches implementation with api.parse.com

Current coverage is 92.24% (diff: 66.66%)

Merging #2721 into master will decrease coverage by <.01%

@@             master      #2721   diff @@
==========================================
  Files           102        102
  Lines         12446      12451     +5
  Methods        1559       1559
  Messages          0          0
  Branches       2039       2040     +1
==========================================
+ Hits          11482      11486     +4
- Misses          964        965     +1
  Partials          0          0

Powered by Codecov. Last update 0773523...ac92e63

@flovilmart @drew-gross when you expect to release 2.2.20 in order for be able to get it via npm?

probably tonight


Auto-update on macOS and Linux

Currently, it still downloads the Windows executable and run it (issue #792), which does not make sense.
If there is no easy way to auto-update the program (without an external binary, that is), the auto-update feature should be replaced by a "Do you want to open the downloads page?" popup.

Should Sparkle be added? Seems like a lot of work for little gain.

Looks like a cool framework, but yeah, probably not worth it.


[Removal]: Chilly Clear Hornet

Hotspot Name
Chilly Clear Hornet
Hotspot b58 Address
112XWbFEfhxA2D5ZHAL9sGpuYMAiWHx1qbqZLUhQ8RbowQY8dfEC
Discord Handle
Darlog#3544
Hotspot Manufacturer
Syncrob.it
Removal Reason
I bought this hotspot recently and I am now selling it to Macedonia, I did nothing to violate any guidelines or rules.
Modifications
No
Extra forwarders
No
Extra antennas
No
Additional Information
I bought this recently and want to sell it to a person I know in M.acedonia. What is the issue? I did not violate any rules. This is happening to 4 of my hotspots!!!
Edit:
After reading other issues I understand that you probably banned my hotspot due to activities that happened before I bought it. I am a secondary market buyer and some of my hotspots have already been resold by me. I bought multiple hotspots to distribute them amongst people I know. I am acting within the community guidelines and I am optimistic that you will come to the conclusion that all of my hotspots should be removed from the list. After all, I am trying to support this community and I am acting responsibly. Thank you.

We have taken into account this removal request in the next iteration of the denylist. It may take some time before it's processed. This is an automated comment due to the volume of addition and removal requests. A future tool is in the works to provide more insight into the analysis approach made by the Helium team and community members that maintain this list.


Slow Upload

Hi,
The download of the emails is very fast, but the upload is very slow.
For 4.5Gb, it took 9 hours. Is that what is expected?
The internet bandwidth is not a problem.

Yes, write is slower than read, that's just how it goes.
Total mail size is not terribly relevant, message count is the more important stat. You can generally expect writing to occur at a rate of one message every 2-3 seconds.
Jay

OK. thank you.


v2.0-rc

NEXT MAJOR release
The following items are to be included with the next MAJOR release of the Auro WC-Generator, all PRs are to be from the feature branch to v2.0-rc branch.

[ ] https://github.com/AlaskaAirlines/auro-header/issues/13

Closing this rc


incorrect warnings for "unused variables" that are actually used

Environment data

Language Server version: 2023.7.10 (pyright 74d8f3c4)
OS and version: Win10 Pro 22H2
Python version (& distribution if applicable, e.g. Anaconda): 3.10

Code Snippet
...
b_step1_getalltenants = args.new_tenant_list
...
sf_tenants_3_settings_sorted= f"{DIR_TEMP}/{s_run_date}_{s_master_tenant}_2_settings[{s_taskid}]_sorted"
...
...
try:
    if not b_step1_getalltenants:
        print(123)
        ld_tenants_todo_flat = json_read(sf_foobar)
except FileNotFoundError:
    print(123)
    try:
        ld_tenants_todo_flat = csv_read(foobar)
    except FileNotFoundError:
        print(123)
        b_step1_getalltenants = True
###########################  FOOBAR ###############################
if b_step1_getalltenants:
...
...
json_save(ld_tenants_done_flat, sf_tenants_3_settings_sorted)

Repro Steps

assign var and use it later

Expected behavior
no pylance warning
Actual behavior
some pylance warnings that should not be there (but only for 10% of the vars)
warning: "b_step1_getalltenants is unused"
warning: "sf_assets is unused"
Logs
I could not reproduce the error in a sample script,

the whole script may contains sensitive information, is a logging needed?```

Could you provide a minimal, self-contained code sample that demonstrates the problem? Your code above references a bunch of symbols that are not defined (e.g.  args and DIR_TEMP), which makes it difficult to repro the problem you're seeing.

Could you provide a minimal, self-contained code sample that demonstrates the problem? Your code above references a bunch of symbols that are not defined (e.g. args and DIR_TEMP), which makes it difficult to repro the problem you're seeing.

hey there,
currently i'm unable to reproduce the behavior in a minimalistic sample.
The Code combinations seems to run pylance unfunctional, the effect disappears when removing parts of the code and cannot be traced down to a certain part of the code.
I understand it's hard to qualify this issues with the example,
I'll come back after checking if I can supply the example or not

Closing awaiting repro.
If this continues to be a problem, would you please reopen the issue and add the information in the report as described in the troubleshooting guide?


Schema generator does not add new indexes when updating

Describe the bug
When running schemaGenerator.updateSchema(), new unique indexes added to the column do not get created.
To Reproduce
Steps to reproduce the behavior:

Create an entity such as Article with a slug string column.
Create the articles table and slug field in the database.
Run schema generator update schema method. Notice the new index is not created.

Expected behavior
A clear and concise description of what you expected to happen.
Additional context

I have only tested this on mysql.
I believe this is the origin of the problem after spending a moment debugging:

// @mikro-orm/knex/SchemaHelper

    hasSameIndex(prop, column) {
        if ([core_1.ReferenceType.SCALAR, core_1.ReferenceType.EMBEDDED].includes(prop.reference)) {
            // TODO: Check indexes on scalar fields instead of returning true.
            return true;
        }
        return prop.referencedColumnNames.some(referencedColumnName => {
            return !!column.fk && referencedColumnName === column.fk.referencedColumnName && prop.referencedTableName === column.fk.referencedTableName;
        });
    }

Versions

Dependency
Version

node
12.17.0

typescript
3.9.5

mikro-orm
4.2.0

your-driver
mysql

Hey @B4nan please could you share an update on this bug ? Would really appreciate it, thanks.

Index diffing happens in findIndexDifference method (hasSameIndex is used only for diffing FK indexes), but that was adding missing indexes only for those defined on entity level (those are stored in meta.indexes and meta.uniques), not on property level (stored on meta.properties[prop].index and unique respectively).


Resolve #Issue-5: HDP2.3 & Ambari 2.1 Support

./gradlew clean dist

produces a new rpm artefact: spring-xd-plugin-hdp23-1.2-2.noarch.rpm
Summary:

Add new build artefact: spring-xd-plugin-hdp23 and build task: hdp23Rpm
Extend the postinstall.sh and preremove.sh to support HDP/2.3 repository structure. NOTE that you could have multiple Stacks installed at the same time. It is not mutually excluding either PHD or HDP or ..
Ambari 2.1 replaces HdfsDirectory by HdfsResource. I've modified to params.py and springxd.py to support the new API and still provide backaward compatibility

Thanks, it sucks that we try to keep same plugin for all ambari versions but then they just rename python methods. I wonder if we should make a tweak to resolve if HdfsDirectory or HdfsResource should be used.
@trisberg do you have comments on this?

ah, ok this PR handles both HdfsDirectory and HdfsResource. Nice!

Spring XD currently supports HDP 2.2 so maybe we should wait with this PR until we update XD to support HDP 2.3

@jvalkeal , indeed i've tried to handle the schizmatic HdfsDirectory/HdfsResource change. I've used the the python function to smoothen the transition and keep backward compatibility. I've tested it on small HDP2.3 cluster and it seems to be working. I've tested both manual (Ambari Wizard) and Blueprint API installations. Here is my Blueprint.
Note that current implementation uses the Stack version to decide which Hdfs API to use. This is suboptimal as the change is Ambari related (e.g from 2.0 to 2.1 - AMBARI-11034) and should be Stack agnostic.

@trisberg this is a valid point. It depends though how long it might take for this update to come. Perhaps in the meantime until the update happens we can map internally HDP2.3 to HDP2.2. Have not tested this yet.

@tzolov I'd rather not go there since we have customers that run a variety of workloads, very difficult to support unless it is officially supported by XD. The plan right now is for XD 1.3 to support HDP 2.3 and PHD 3.3 some time after Spring One which is in September.

Fair point

ok and kafka config breaks with hdp2.3, so it needs to be fixed first anyway.

closing this pr. @tzolov I picked related changes from this pr to fix for #5, thx mate!

@jvalkeal YW


Add datetime helper functions

👓 @spladug
Extending the datetime module with functions that we keep rewriting in each micro service

Would it also be useful to include a utcnow function that returns a the current time, in UTC, with a timezone set?

💇
Changes:

Add get_utc_now function
Add type annotation to functions
Rename module from chronos to datetime
Use datetime timezone instead of pytz timezone

💇 @pacejackson
Change:
Using floor division now


chore: formatting and readability

Good job on the list.
Improving the formatting as it's a little hard to read outside of the markdown preview.
I replaced the inline links with references and added some blockquotes for your citations and notes.
I also experimented with splitting the table into 3 sections to improve the immediate readability. Not sure what you think, it's a little opinionated but that should be a good change overall.

Good changes.


docs(integrations): remove duplicate docstring in WandbModelCheckpoint callback

Fixes #5241
Checklist

[x] Include reference to internal ticket "Fixes WB-NNNN" and/or GitHub issue "Fixes #NNNN" (if applicable)
[x] Ensure PR title compliance with the conventional commits standards

Thanks.. this was fixed in:
https://github.com/wandb/wandb/pull/5289


Improve validation speed for large number of files

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Do not need to list all files in GCS for validation. Add limit field to fileIO

R: @chamikaramj PTAL

LGTM. Thanks.

@davorbonaci please merge.
Thanks you.

Conflicts. Please rebase on top of the current python-sdk branch.

@davorbonaci fixed now. Thanks again.

@davorbonaci ping for merge

Can merge, if @davorbonaci or @robertwb hsa not.
But: is there a JIRA issue for this change?

@dhalperi that would be great. There is no JIRA for this right now.

Please file it & update the title.

@davorbonaci done, thanks.

Merged, thanks.

Thanks


Amp, Ubuntu

Bug Report
System Information

linux, Ubuntu 18.04.5 LTS
v2.0.9, built 04/01/2021 22:03
Main

I confirm:

[x] that I have searched for an existing bug report for this issue.
[x] that I am using the latest available version of AMP.
[x] that my operating system is up-to-date.

Symptoms

Trying to resolve a process that starts up once or twice a day, then hangs using a large amount of cpu usage. Under KSSwap, this process should do what it needs to do i.e. move certain inactive data to the swap drive then terminate or idle. once running it does not end and has to be manually stopped. This of course causes impact on the servers that are running with additional lag and minecraft tps issues
Reproduction

Unclear how to reproduce this one, as not 100% sure its an amp issue, as i havnt seen many if any bug reports relating to the same thing. Will be doing a clean install of amp once can arrange external backup servers to run, as this instance is many updates in.

Unrelated to AMP, you should check syslog/dmesg to see what's going on.


Don't assume a single emission in AutoPersistedQueryInterceptor

Resolves #5667

There's a bunch of "interesting" edge cases if the PersistedQueriesNotFound is followed by other items in the original Flow. I don't think we need to be too concerned but mentioning just in case you think of something?

Yeah I think it should "never" happen (with quotes because odd things tend to happen 😅). Worst case, it will re-query with the body even if it's not strictly necessary - I think that's OK.


[Improvement] delete star chart

the star chart for stargazers over time can't show normally, so I delete it

Codecov Report

Merging #13953 (8cba326) into dev (0a5b32e) will decrease coverage by 0.01%.
The diff coverage is n/a.

:exclamation: Current head 8cba326 differs from pull request most recent head a7c534d. Consider uploading reports for the commit a7c534d to get more accurate results

@@             Coverage Diff              @@
##                dev   #13953      +/-   ##
============================================
- Coverage     38.90%   38.89%   -0.01%
- Complexity     4469     4474       +5
============================================
  Files          1164     1164
  Lines         42444    42444
  Branches       4761     4757       -4
============================================
- Hits          16511    16510       -1
  Misses        24114    24114
- Partials       1819     1820       +1

see 1 file with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


unable to start the UI

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/2446
Originally created by @kalmenchia on 2020-11-11 15:59:28

Hi,
i have tried to setup Meltano, before i setup the .env , i can access using the HTTP://<mydomain.com>:5000, but after i did the
following ,

meltano config meltano set ui server_name meltano.example.com
meltano config meltano set ui session_cookie_domain meltano.example.com
meltano ui setup meltano.example.com
meltano config meltano set ui forwarded_allow_ips "*"
meltano user add admin securepassword --role admin
meltano config meltano set ui authentication true
set the .env to 644 permission ( if not doing so, my meltano.service won't start)

i restart the meltano UI , i got the error ,

The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

can anyone give a hand for some direction please, thanks.

View 7 previous comments from the original issue on GitLab


training one subject, then training anothjer one on same model while preserving identity of both

is that even possible?
I trained a woman for 3k and 30 pics, came out great, so  and it gave me ckpt that i used to train a man as new subject, but in the end the woman starts to look like that new subject - a man, this is really an issue to train multiple people with same model, so the only way is to do it all of them at once ? How to preserve the prrevious subjects when training new ones ?

keep a few instance images of the first person in the instance folder.

ok, trained 2 subjects, now put back 3  head shot images of 2 previous subjects and training 3rd subject on new 30 imgs , we will see how it will come out

OK i think it works but it overfits previous subjects with 3 images and i kinda want to freeze their training...

For example, if you're training a subject with 15 pics, train 200-300 steps with the " --train_text_encoder " argument, the resume the rest of the training without it (3000 steps), you'll get better style transfer and less overfitting. I'll add tat option shortly

i resumed training and deleted 2 pics of first subjects  so theres just 2 pics per each, this is not enough it starts to lose identity when training even 500 steps and their identity was fine without additional training, so each new training starts to los eidentity , ill try without text encoder.
What im tryin to do really is to "fix " some subjects like actors , and train them for awhile , i thought you cn just train one actor for like 3k steps then start next actor with 3k steps and youll get two nicely trained subjects but with each training the whole latent space is leaking with new subject... is that even solvable ?

you know that the dreambooth script has a problem of bleeding meaning every time you train a subject, it bleeds to others in the same class

yes i know so there is no remedy? or ability to tone down trainig for subjects that were already trained ?

cutting the text encoder training steps short helps

ok i just ran atest - 16 subjects, 4 images per subject, 5000 steps, its gonna be a shitshow or its gonna work just fine, if not, ill try with text encoder disabling

make sure you run with the text encoder 200 steps per subject

where do i change that? im on defaults but im 50% into training so i can try it when this one finishes

it trains randomly, you run the training for 200x16 with the text encoder, then remove the argument --train_text_encoder, after that resume training for the rest of the steps

Ok some subjects overfit, some wherent there yet, theres morevariables to it, the ones that were already in SD overfit faster cause of resemblance, and the ones that are new barely pickedup likeness, theres just no way to get reasonably good results with so many moving parts out of our control, i did it as a test what would happen and its pretty mucjh what i got from other multimodel trainings, you can do it with 2 mayb 3 subjects but its still a risk one wont be there yet.
Until theres no way to stop other subjects from training , its just not going to work that well with like 10 people at once, but id be happy to be proven wrong.
Maybe loss detection and extra code to skip the ones that have already good likeness would help, but thats just theory.

in a few minutes there will be a feature to limit the text_encoder aggressive training

great , i also suspect that 2e is too high to train multiple subjects with 3000 steps  they pretty much are on border of overfitting and that rate it gets great likeness but also suffers when stylising into non photo related modes like comics , but thats easy to change by myself

added, use https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb
Start with 10% and avoid to many subjects, keep it below 3.


[components] Create custom dialog

Current Behavior

Desired Behavior

Implementation

Acceptance Tests

Mockups

Contributor Guide

🎨 Wireframes and designs for Layer5 site in Figma (open invite)

is there any guide how to start till project locally

This is done here: https://github.com/layer5io/sistent/tree/master/packages/components/src/custom/Dialog


v0.1.0

Anything we wanna do prior to a first release? :)

Yolo, let's do v0.0.1 with what we've got :) and talk about v0.1.0.

Yolo, let's do v0.0.1 with what we've got :) and talk about v0.1.0.

v0.0.2 landed in no-time :D

Maybe finalizing the included libraries list and adding an example for each is enough for our first minor release?

So first of all, It's all pretty arbitrary :) it would probably be fine.
But ... one thing to keep in mind is, we don't know yet how the scala-cli integration may work. It may require us to publish under a different artifact name (I heard some murmur of just "kit" instead of "toolkit" for example).
Another thing to keep in mind is that although I believe Scala Toolkit will be going to 1.0.0 in the near future, we technically won't be able to until http4s goes to 1.0.0. So that might be a while yet.

I propose we release v0.1.0 to coordinate with the announcement blog post.

https://github.com/scala/scala-lang/pull/1512

https://github.com/typelevel/toolkit/releases/tag/v0.1.0


Svg offset deprecated

Hi!
https://www.dropbox.com/s/xfrdijcf2z7nxfy/Скриншот 2016-01-30 20.22.36.png?dl=0
I use Snap.svg in my project.
Today in google chrome I saw caution about SVGElement.offsetHeight \ Width methods.
I fix in with this (https://www.chromestatus.com/features/5724912467574784) recommendations.
I hope this will be helpful.
Philipp.

:+1:

Any news on this?  Is it ok to merge?


Swarm flavor revamp

This PR overhauls how the Swarm flavor plugin works.  Before, the swarm flavor had to be run separately between the manager and workers -- where the manager is a single node, configured by a vanilla plugin, while the workers are configured separately using the swarm flavor itself.  Now the swarm flavor plugin

relies heavily on templates - one for worker and one for manager to give the user the ability to customize what additional tasks are to be run in the init scripts.
uses  the manager's multi-group handling to make managing two different groups in a single JSON configuration possible.
the templates allows the user to control the init for the first node (where swarm init is needed) and the manager followers and the workers.  New template functions are introduced so that the user can essentially program how the init sequence for the different nodes should work.
removes the need to bootstrap initial leader manager node via the vanilla plugin.
a single configuration, a single commit, can provision both the manager and worker groups.
a different version of using actual get.docker.com script to install Docker is also provided -- as compared to the previous demos of using pre-installed docker on vagrant boxes.

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "swarm-flavor-revamp" git@github.com:chungers/infrakit.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842354041880
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.


"The above error occurred in the  component: ..." after successful init

Created project with prisma2 init foo: Blank project > SQLite > Confirm > Javascript > Demo script, got success screen but also an error:
C:\Users\Jan\Documents\throwaway
λ prisma2 init 617_2

 SUCCESS  The 617_2 directory was created!
 SUCCESS  Prisma is connected to your database at file:dev.db

    Navigate into the project directory:
    $ cd 617_2

    Start Prisma's development mode to enable access to
    Prisma Studio and watch schema.prisma for changes:
    $ prisma2 dev

    Start the REST API server (in a new terminal window/tab):
    $ npm run dev

    Learn more about the Prisma Framework:
    https://github.com/prisma/prisma2/

    If you encounter any issues, please report them here:
    https://github.com/prisma/prisma-examples/issues/new

The above error occurred in the <Step61Success> component:
    in Step61Success
    in Unknown
    in Unknown
    in TabIndexProvider
    in div (created by Box)
    in Box
    in Unknown
    in App

React will try to recreate this component tree from scratch using the error boundary you provided, App.

C:\Users\Jan\Documents
λ prisma2 -v
<EMAIL_ADDRESS>binary version: 8d88815bac0dd673ab8c2def11f441ef44fb189d

Hi,
Thanks for reporting this! It has already been reported before, so I am going to close this as a duplicate of https://github.com/prisma/prisma2/issues/1085


Reolink-RLC 822A

hi there,
i use this code for the Reolink Camera and the "stillImageSource"
after i changed from http// to https// the stllimage works perfekt but the rest doesn't
(see the code at the end).
i get this error message
24/01/2022, 22:08:35] [CameraUI] CAMERA NAME: FFmpeg exited with code: 1 and signal: null - [tcp @ 0x322cac0] [error] Connection to tcp://<IP_ADDRESS>:554?timeout=0 failed: Connection refused - [error] rtsp://USER:XXX@<IP_ADDRESS>:554/h264Preview_01_main: Connection refused
[24/01/2022, 22:08:35] [CameraUI] CAMERA NAME: Stopped video stream.
and here is the code:
"name": "CAMERA NAME",                    "motion": true,                    "motionTimeout": 15,                    "unbridge": true,                    "hsv": true,                    "prebuffering": true,                    "prebufferLength": 4,                    "videoConfig": {                        "source": "-rtsp_transport tcp -re -i rtsp://USER:XXX@<IP_ADDRESS>:554/h264Preview_01_main",                        "subSource": "-rtsp_transport tcp -re -i rtsp://USER:XXX@<IP_ADDRESS>:554/h264Preview_01_main",                        "stillImageSource": "-i https://<IP_ADDRESS>/cgi-bin/api.cgi?cmd=Snap&channel=0&rs=wuuPhkmUCeI9WG7C&user=XXX&password=XXX",                        "maxStreams": 2,                        "vcodec": "copy",                        "acodec": "libfdk_aac",                        "audio": false                    },                    "smtp": {                        "email": "CAMERA NAME"                    },                    "videoanalysis": {                        "active": true                    },                    "mqtt": {}
thank you in advance georgios

Hi,
As a RLC 822A owner I can help 🙂 this camera is a 4K one so the main stream doesn't support h264 but h265. Since camera UI currently only support rtsp over h264 here is the configuration you need to apply in your videoConfig:
"videoConfig": {
                        "source": "-i rtsp://user:password@<IP_ADDRESS>:554/h264Preview_01_sub",
                        "subSource": "-i rtsp://user:password@<IP_ADDRESS>:554/h264Preview_01_sub",
                        "stillImageSource": "-i  https://<IP_ADDRESS>/cgi-bin/api.cgi?cmd=Snap&channel=0&rs=wuuPhkmUCeI9WG7C&user=XXX&password=XXX",
                        "rtspTransport": "tcp",
                        "maxStreams": 2,
                        "vcodec": "copy",
                        "acodec": "libfdk_aac",
                        "stimeout": 10,
                        "audio": false
                    }

Please adopt the config as described above

i've done it but still doesn't work. i'm gonna try something else and i will come back to you. thank you georgios

If you have the latest firmware version v<IP_ADDRESS>4_21121707 or above (for users in the future) you need to enable RTSP.
Simplest way is to use the web interface using https https://<your camera ip address> then:

click on settings (top right gear icon)
click on Network Settings
Port Settings (yes that's somewhat related to ports)
Enable RTSP

I had this issue yesterday after updating my cameras 🙂

THANK YOU!!!
it's working you are great!
georgios_


Geoposition closest petrol station

Hello all,
Hello HGlab01,
I am not sure,
if I am doing something wrong
or something is wrong in the creation of datapoints for the closest petrol station.
For example I set the following Geopositions, which I got from https://www.gps-coordinates.net/.
Gaspoltshofen-Benzin
Gaspoltshofen-Diesel
48.134976346709536
13.728116446595454
=> this should be "Aspöck - Freie Tankstelle"
Ampflwang-Benzin
Ampflwang-Diesel
48.08074897870017
13.563780273867803
=> this should be "avanti - Ampflwang Siedlung 170"
Frankenburg-Benzin
Frankenburg-Diesel
48.06979710140276
13.492110726209582
=> this should be "AVIA Xpress (Automaten-Tankstelle)"
Zipf-Benzin
Zipf-Diesel
48.025970359863884
13.510162524555032
=> this should be "Lagerhaus Zipf"
But in the objects it creates mostly always the station "AVIA Xpress" as the first station,
which is the nearest to my living place (geo position from iobroker settings).
The problem is that I dont get the price, because the stations I want are
mostly on the 5th place or higher, so there won´t be created a price datapoint.
It is just created a price datapoint object for the first five or also called 0,1,2,3,4 datapoints.
Can you help me with this issue?
Many thanks in advance for any respons.
Greetings Cyberraph (iobroker-newbie) :-)
PS:
As I see the prices for fuelpricemonitor are from e-control.
I have a ÖAMTC App, where prices are also displayed.
These prices also come from e-control.
In this App I saw that sometimes prices disappear.
So I guess also in io-broker there will be sometimes a price and sometimes no price displayed,
when io-brokes updates the datapoints and e-contro
l does´nt offer prices at this particular day-time / update-time.
Pictures from my io-broker fuelpricemonitor

Hi!
I guess it is, because there are always 5 others cheaper than the one you would like to know within the closest 10 stations?
You can use
https://www.spritpreisrechner.at/#/fossil
to understand better the result and maybe with this you can optimize the coordinates
For Gaspoltshofen it's obvious that "Aspöck - Freie Tankstelle" is not cheap enough within the closest 10 stations

Hello HGlab01,
oh, I am so sorry.
I thought it creates datapoints for the closest stations according their position next to the set geoposition.
And not the next 5 which have the cheapest prices.
That was not clear enough for me as I read the descriptions.
And in the adapter settings -> additional settings -> "Show cheapest staions over all locations"
just creates additional datapoints to compare the set geo-points within their cheapest stations themselfs.

Now I understand it.
Therefore the issue was sitting in front of the PC in this case.
And I have to search for another solution to get just my wanted stations.
No matter, if they are the cheapest or not.
I just want to compare the mentioned stations.
But I dont see to get it with this adapter, as the stations I want will sometimes fall out of the 5 cheapest.
Or I can make it happen to set a geo position to get what I want.
Thank you.
You have helped me enough.


when will the code be released? thx~

I have the same question

me too

Unfortunately, we have some delay in releasing the code. We sincerely regret the inconvenience. We have the updated the readme.

Thank you so much for the patience. The preliminary version for the imagenet dataset is available in the imagenet folder.


Function literal signature lost when passing around

Okay the title is suboptimal, but meh.
Setup:
lib LibFake
  alias Faker = Void* -> Void
  fun fake(faker : Faker)
end

module Fakes
  def self.main_fake data
  end
end

def fake faker
  LibFake.fake faker
end

def strict_fake faker : LibFake::Faker
  LibFake.fake faker
end

fake ->Fakes.main_fake

gives
Error in ./tmp.cr:20: wrong number of arguments for 'Fakes:Class#main_fake' (0 for 1)

fake ->Fakes.main_fake
     ^

strict_fake ->Fakes.main_fake

still gives
Error in ./tmp.cr:20: wrong number of arguments for 'Fakes:Class#main_fake' (0 for 1)

strict_fake ->Fakes.main_fake

When the type is specified in the wrapper, it should also work when overloads for Fakes.main_fake exist.

Marked as enhancement, but this works "as expected". We find matching defs based on the type of the arguments, except when it's about C functions (because they can't be overloaded and you must specify their types, so we can infer argument types there).
I'll leave it open in case we do find a way to improve this.

I'll close. We find matching defs from the argument types, so there really isn't a way to do it the other way around.


Uncaught (in promise) ReferenceError: require is not defined

Getting issue in console on production(npm run build).

I got this also. I think it is due to dayjs in production build uses require(cjs) instead of import(mjs)

I used requirejs package also via cdn. But i got other issues.

@srini-leanfolks I moved the package out of node modules and fixed this part for myself.
in index.esm.js you can replace require to import isBetween from 'dayjs/plugin/isBetween (on line 940)
and i commented switch case (on line 150)
Waiting for when will be fixed this part. :)

@srini-leanfolks how it gets updated in npmjs.com ?

@srini-leanfolks how it gets updated in npmjs.com ?

Sorry I didn't get you

Fix in v1.4.1
Closing this issue

I'm still getting the same error in production builds. Using version 1.4.1. (dayjs v1.11.6)


Clean up Registry.Auths code

What is the problem you're trying to solve
https://github.com/containerd/containerd/blob/878132923db35f31c8c7b324d9cd1e5f59046064/pkg/cri/config/config.go#L206-L207
Describe the solution you'd like
Remove Auths relevant code.
Additional context
No response

https://github.com/containerd/containerd/pull/7526 looks like it's doing this


Use ServiceWorkers to make this app available offline

Itty-bitty could then be used as micro-web-app loader.
We could open any URL without the need of an internet connection.

Currently appcache is enabled for the root file. I know this is deprecated (and has issues), but it should provide the same behavior. We can investigate serviceworkers later if needed.


[$1000] Empty chat header when starting a chat with any email containing captial letter

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Go to the app and click the Search icon in the LHN.
Type an email that is not in your contact list and make sure that the email contains an uppercase letter ie<EMAIL_ADDRESS>Select this contact.

Expected Result:
The chat header should show the email and not be empty.
Actual Result:
Chat header is empty
Workaround:
Can the user still use Expensify without this being fixed? Have you informed them of the workaround?
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.3.31-2
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation

https://github.com/Expensify/App/assets/43996225/1b90542d-36a9-4f0d-8bb7-5be5e4872131
Expensify/Expensify Issue URL:
Issue reported by: @allroundexperts
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1687051826855649
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~0139d93d50b04a4799
Upwork Job ID:<PHONE_NUMBER>378838016
Last Price Increase: 2023-06-26

Proposal
Please re-state the problem that we are trying to solve in this issue.
Empty chat header appears when starting with a chat with any email containing capital letter
What is the root cause of that problem?
Weird message returns when email has capital letter as follows
{
  "onyxMethod": "merge",
  "key": "report_6157911605865567",
  "value": {
    "reportID": "6157911605865567",
    "reportName": "Chat Report",
    "type": "chat",
    "chatType": "",
    "ownerEmail": "__fake__",
    "ownerAccountID": 0,
    "policyID": "_FAKE_",
    "participants": [],
    "participantAccountIDs": [],
    "isPinned": false,
    "lastReadTime": null,
    "lastMentionedTime": null,
    "lastReadSequenceNumber": 0,
    "lastVisibleActionCreated": "2023-06-23 21:30:51.301",
    "lastVisibleActionLastModified": "2023-06-23 21:30:51.301",
    "lastMessageText": "",
    "lastActorEmail": "__fake__",
    "lastActorAccountID": 0,
    "notificationPreference": "always",
    "welcomeMessage": "",
    "stateNum": 0,
    "statusNum": 0,
    "oldPolicyName": "",
    "visibility": null,
    "isOwnPolicyExpenseChat": false,
    "lastMessageHtml": "",
    "hasOutstandingIOU": false,
    "policyName": null,
    "hasParentAccess": null,
    "parentReportID": null,
    "parentReportActionID": null,
    "writeCapability": "all"
  }
}

As you can see above, empty participants is returned. This makes the report title empty string and shows nothing because of this line
https://github.com/Expensify/App/blob/96e534a5b28cecb7cb221986a2dbca57c6d2fc1e/src/pages/home/HeaderView.js#L158
What changes do you think we should make in order to solve the problem?
This issue looks like a backend issue but can have a workaround.
We can remove response data with keys staring with report_ and value with lastReadTime is null(check the above response) before the below line
https://github.com/Expensify/App/blob/96e534a5b28cecb7cb221986a2dbca57c6d2fc1e/src/libs/Middleware/SaveResponseInOnyx.js#L24C120-L24C120
This works as expected.

Result
https://github.com/Expensify/App/assets/126839717/71a607ba-9edf-40dd-8bba-629d659907bc

What alternative solutions did you explore? (Optional)

Reproducible, triaging to external

@rushatgabhane we have some proposals, what do you think?

@rushatgabhane
Yes. First, we received the message with valid participantAccountIDs, but please check the later items in the onyxData array. It has the report item as well and it has empty array participantAccountIDs. So if all of them are merged, participantAccountIDs would become an empty array. That's why I suggest to filter out the invalid onyxData items.

the OpenReport API returns an empty array for participantAccountIDs

I agree this seems like the root cause and would need a backend fix.

we also optimistic store the email with capital letter in personalDetailsList

@Beamanator @puneetlath can you check this out? I think we are about to remove this part of the frontend?

@flodnv

we also optimistic store the email with capital letter in personalDetailsList

@Beamanator @puneetlath can you check this out? I think we are about to remove this part of the frontend?

We removed personalDetails, we're not removing personalDetailsList
I think this will require both back & front-end fixes, I can take a look at the backend part tomorrow if you want @puneetlath

Web PR is up - https://github.com/Expensify/Web-Expensify/pull/38069
@rushatgabhane I think we agreed we need to also have an App PR which lowercases emails when storing them in Onyx? Are we ready to review proposals here? Or do we want to wait till the backend PR is on staging?

@s-alves10 by "fixed in the backend" do you mean "allow mixed-case emails everywhere"?

We can do all email comparison in lowercase. I think users should see what they input. Email is not case sensitive and there is no side effects.

Sadly in or backend email is case sensitive 😬  And we expect it to be lowercase...

I put a backend fix in place which is now in staging, but not in production yet - but i still think we need to auto-lowercase the optimistic emails we set when creating the new chat

Can we get new proposals please. Let's fix this close to where we're creating an optimistic report.
I don't like the solutions proposing a fix in search page.

@sonialiap can you confirm it's fixed and if so, close the issue?

Tested, confirming that this issue has been fixed


updated index.html, added a new file navbar.css

Description:

Explain the functionality or bug fix implemented in this pull request, including its purpose and significance.

Changes Made:

Provide a detailed overview of the code changes made to introduce this implementation, along with any relevant code snippets or files.

Related Issue:

Fixes #[ISSUE_NUMBER]

Screenshots (if applicable):

Attach screenshots, if applicable, to showcase the newly implemented feature or bug fix.

You are suppose to make a Pull Request to home-page branch instead of merging to main

your list of nav links contain home, about, contact, portfolio, team, blog instead of  home, about, contact, event, project, blog

And also i dont think the js file is neccessary as all the file we are to use as been added as a template, the onhover drop down you also add to some of the link, i dont also see that in the design


Integrate filter outputs into evaluation utility

Evaluation now prints a new column for all filter-generated inputs and outputs.

FWIW, I don't think we use the evaluation binary much anymore. It's not used for the AEC eval. We instead grep on the output the the regular synthesis command. I'm not sure this is really needed. But I don't oppose fixing the evaluation binary.


When the drop database xxx force is executed, the storage directory is preserved

Steps to reproduce the behavior (Required)
drop database xxx force

Expected behavior (Required)
Recursively delete the directory corresponding to the HDFS file system
Real behavior (Required)
The table data in the corresponding HDFS file system is deleted, but the folder is retained

StarRocks version (Required)
Use HDFS for shared-data 3.2.8

please check if there is any files left under the directory.
It is a little bit different between HDFS and OBJECT storage. For object storage, when all the files are cleaned, the directory will be gone (because there is no directory at all), but for HDFS, directories are in namenode meta as well. Currently there is no explicit clean of such directories.

The table data in the corresponding HDFS file system is deleted

So it's clear that the corresponding folder Use HDFS for shared-data 3.2.8 will not be deleted, right?

The table data in the corresponding HDFS file system is deleted
So it's clear that the corresponding folder Use HDFS for shared-data 3.2.8 will not be deleted, right?

Yes, and it is expected for now when using HDFS as its remote storage.

@yesengh Kevin is right, db dir can not be deleted for now, maybe you can remove it manually

OK, we can manually clean up this kind of low frequency problem, thank you.