Loading samples to RAM with DDP.

❓ Questions and Help
What is your question?
I'm facing an IO bottleneck that can be fixed with a custom torch.utils.data.Dataset that loads each sample to RAM. Then training goes fast as I don't need to read my samples (images) from disk (slow). Everything works well with when I'm using 1 GPU, but I'm a bit lost when I switch to multiple GPUs with DDP.
DDP divides the samples to each GPU and I'm wondering when/where I should load my samples to RAM so that each process only loads its own partition of the samples?
Code
class RAMDataset(data.Dataset):
    def __init__(self, paths,labels,transform):
        """Dataset that loads all samples to RAM."""
        self.paths = paths
        self.labels = labels
        self.transform = transform

    def __len__(self):
        return len(self.samples)

    def load_to_RAM(self):
        self.images = []
        for path in self.paths:
            with open(path, "rb") as f:
                str_encode = f.read()
                nparr = np.frombuffer(str_encode, np.uint8)
                self.images.append(cv2.imdecode(nparr, cv2.IMREAD_COLOR))

    def __getitem__(self, index):
        # Run self.load_to_RAM() first!
        image = self.transform(self.images[index])
        label = self.labels[index]
        return image, label

What have you tried?
With 1 GPU self.load_to_RAM() can be excecuted as soon as the Dataset has been created.
dataset = RAMDataset(paths,labels)
dataset.load_to_RAM()
loader = DataLoader(dataset,...)
trainer.fit(model,loader)

But obviously this would load the samples num_gpus times to the RAM of the node.
I quickly tried to call self.train_dataloader.dataset.load_to_RAM()on the hook setup() but got the following error...
AttributeError: '_PatchDataLoader' object has no attribute 'dataset'

..and I'm 99% this solution would also load all of the samples to RAM.
Possible solution?

Find out which process (which GPU and which node) is currently running.
Get the allocated slice of the samples for this process.
Load only that slice of the self.paths to RAM.

Tried to go through the source code but couldn't find out how I could implement this.

The OS should automatically cache the dataset after the first epoch (assuming you have enough ram) so you shouldn't need to do this.
I'm actually not sure if distributed sampler shards the same data/item on the same GPU through different epochs... but that's more of a pytorch question than a lightning one.

Lightning uses a random Sampler to extract data from the Dataset. No easy way to know in advance the indices that will be used on each machine. However, you can create a Dataset where only the right portion of the data is loaded into RAM before training on each GPU/node. You can use torch.distributed.get_world_size() and torch.distributed.get_rank() to know which data should be loaded on each parallel process.
Example (I did not test it):
class MyDataset():
    def __init__(self, data):
        start = (len(data) // torch.distributed.get_world_size()) * torch.distributed.get_rank()
        end = (len(data) // torch.distributed.get_world_size()) * (torch.distributed.get_rank() + 1)
        self.data = load_ram(data[start:end])

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        # checks on index ...
        return self.data[idx]

I do have enough RAM if I manage to divide my data to multiple nodes. If the BasicDataset() below is cached after the first epoch wouldn't that mean only the labels and paths are cached? What I would want is that the loaded images are cached. I'll look into PyTorch and see if the same data/items are kept for each epoch.
class BasicDataset(data.Dataset):
    def __init__(self, paths,labels,transform):
        self.paths = paths
        self.labels = labels

    def __getitem__(self, index):
        image = load(self.paths[index]) # This is my bottleneck.
        label = self.labels[index]
        return image, label

Ah you're doing cross node DDP, I was referring to single node above... I'm afraid I don't have much experience with multi node DDP.
But if each node gets the same subset of data every epoch, the OS should cache the files all the same (without needing load_to_RAM) after the first epoch since you have enough RAM.

Would the actual loaded images be cached even though they are loaded inside __getitem__() and not just the whole Dataset object with only the paths to these files?

Yes, this is not stored in your program/script memory but instead it's simply file caching at the system level.
I use Ubuntu but this should be the same for other linux ditros unless some memory/caching settings have been changed/limited. You can look at the cache part of your ram usage in htop as your data is loaded into memory during the first epoch.

Perfect! Kept running tests that ran only for one epoch so I never noticed the seed-up on the second epoch... 😄 Well you learn something new everyday.


Blank Map

I am getting a blank map while rendering using React-native-maps. I am able to view the map on first scene after which I am going to another scene where I am encountering this error. I am using geolocator for finding current location.
React-native-router-flux 3.38.0
React-native-maps 0.13.0
React-native 0.42.0

I have same problem with version of 0.12 and
React-native-router-flux 3.38.0
React-native-maps 0.13.0
React-native 0.43.0

Hi,
I added a style to the Map component:
<MapView
  style={styles.map}...
/>

map: {
  position: 'absolute',
  left: 0,
  right: 0,
  top: 0,
  bottom: 0
}

Which displayed the map for me on:
"react": "15.4.2",
"react-native": "0.42.3",
"react-native-maps": "^0.14.0",

Can you show us code, especially with Actions.map() or something :)

Hello,
same problem here with React-native-maps 0.13.0 and React-native 0.42.0
We're using Ignite 2 and add map with : ignite add map which configure the react-native-maps package in the background.
We trying to solve this issue (by updating package, modifying android configs, etc...) we let you know if we find something.
Here is our MapComponent.js file :
`
import React from 'react'
import MapView from 'react-native-maps'
import MapCallout from './MapCallout'
import Styles from './Styles/MapStyles'
import { calculateRegion } from '../Lib/MapHelpers'
export default class MapComponent extends React.Component {
constructor (props) {
super(props)
const locations = [
{ title: 'Location A', latitude: 37.78825, longitude: -122.4324 },
{ title: 'Location B', latitude: 37.75825, longitude: -122.4624 }
]
// const region = calculateRegion(locations, { latPadding: 0.05, longPadding: 0.05 })
const region = { latitude: 123, longitude: 123, latitudeDelta: 0.1, longitudeDelta: 0.1}
this.state = {
region,
locations,
showUserLocation: true
}
this.renderMapMarkers = this.renderMapMarkers.bind(this)
this.onRegionChange = this.onRegionChange.bind(this)
}
componentWillReceiveProps (newProps) {
	this.setState({
	  region: calculateRegion(newProps.locations, { latPadding: 0.1, longPadding: 0.1 })
	})
}

onRegionChange (newRegion) {
	// const searchRegion = {
	//   ne_lat: newRegion.latitude + newRegion.latitudeDelta,
	//   ne_long: newRegion.longitude + newRegion.longitudeDelta,
	//   sw_lat: newRegion.latitude - newRegion.latitudeDelta,
	//   sw_long: newRegion.longitude - newRegion.longitudeDelta
	// }
	// Fetch new data...
}

calloutPress (location) {
	console.tron.log(location) // Reactotron
}

renderMapMarkers (location) {
	return (
		<MapView.Marker key={location.title} coordinate={{latitude: location.latitude, longitude: location.longitude}}>
			<MapCallout location={location} onPress={this.calloutPress} />
		</MapView.Marker>
	)
}

render () {
	return (
		<MapView
			style={Styles.map}
			initialRegion={this.state.region}
			onRegionChangeComplete={this.onRegionChange}
			showsUserLocation={this.state.showUserLocation}
		>
			{this.state.locations.map((location) => this.renderMapMarkers(location))}
		</MapView>
	)
}

}
`

Ok, we finally managed to make it work !
Here's few changes we made :

update RN-maps from 0.13.0 to 0.14.0
in settings.gradle update android path to : react-native-maps/**lib**/android
in your component style make sure you have this :
container: {   ...StyleSheet.absoluteFillObject,   height: 400,   width: 400,   justifyContent: 'flex-end',   alignItems: 'center', }, map: {   ...StyleSheet.absoluteFillObject, },
and the most important : make sure your GOOGLE API KEY  is NOT restricted to android application. You have to set NO RESTRICTION !

Hope this will help.

@AlexandrePalo Sorry for late reply. I was away. I resolved it by passing type:'reset' in flux action. Thanks.

i have a blank maps too
my styles :
  mapContainer: {     ...StyleSheet.absoluteFillObject,     height: 400,     width: 400,     justifyContent: "flex-end",     alignItems: "center"   },   map: {     top: 0,     bottom: 0,     left: 0,     right: 0,     position: "absolute",     ...StyleSheet.absoluteFillObject   }
my package.json :
	{...    	  "dependencies": { 		"react": "16.0.0-alpha.12", 		"react-native": "0.45.1", 		"react-native-camera": "git+https://github.com/lwansbrough/react-native-camera.git", 		"react-native-camera-roll-picker": "^1.1.9", 		"react-navigation": "git+https://github.com/react-community/react-navigation.git", 		"react-native-maps":"0.15.3" 	  }, 	  "devDependencies": { 		"babel-jest": "20.0.3", 		"babel-preset-react-native": "2.0.0", 		"jest": "20.0.4", 		"react-test-renderer": "16.0.0-alpha.12" 	  }    	}
and my js :
import MapView, { MAP_TYPES } from 'react-native-maps';
export default class Test extends Component {
  constructor() {
	super();

	this.state = {
	  region: {
		latitude: 37.78825,
		longitude: -122.4324,
		latitudeDelta: 0.0922,
		longitudeDelta: 0.0421
	  }
	};
  }

  onRegionChange(region) {
	this.setState({ region });
  }

  render() {
	return (
	   <View style={styles.mapContainer}>
		<MapView
		  style={styles.map}
		  region={this.state.region}
		  onRegionChange={this.onRegionChange}
		/>
		<Text style={styles.title}>{this.state.region.latitude}  / {this.state.region.longitude}  </Text>

	  </View>
	);
  }
}

any idea please ?

I have same problem.Map is blank event not showing google logo.
I'm using react-native-map.


Disconnected Environments and "mirroring to a location you control"

Is there an existing issue for this?

[X] I have searched the existing issues

What did you expect to happen?
Hey,
When working in disconnected environments it is normal to have mirror registry somewhere. But this somewhere is normaly also a restricted location which won't allow free unrestricted internet access. As I conduct from multiple closed Issues and your Docs. There is no real answer to solve this problem. The only solution we got is: Don't restrict internet access for at least one endpoint in your environment. This cannot be our allready provided mirror registry. Because we need to secure it. Instead we would need to implement another Registry, outside of our network without any restrictions. And use it as a proxy registry endpoint. The moment this one is not restricted. We could import everything from there. Which in turn makes the whole restriction pointless.
The only two solution I see right now:

Use my private PC to pull Images we need from registry.k8s.io and push them to docker hub or quay, then allowing our mirror Registry to pull from certain repositories in this registries.
Use my private PC and copy it directly into our registries

I believe doing hacky workarounds to get your images into a secure environment devalues the great service you provide and no sane and liable person should do this. Also having a "pull from everywhere" mirror registry does not seem to be a secure choice either.
I was hesitant to even bring up the topic, since your service is not responsible for our needs. In the end I decided to start this discussion again, since alot of kubernetes-sigs images are officially and foremost on this registry. With the high impact of Kubernetes and the neccessity of trusting images and image registries I cannot be the only one who needs a trusted endpoint for this registry.
You mentioned here participating Vendors sometimes offer your Repositories in their Licence model. Which is not the case for the images we need.
Maybe I am missing something! I would love to hear feedback and suggestions, that are feasable for secure and disconnected environments that do not require to create a man-in-the-middle registry or carry important images with an USB stick to their destination.
Kind regards.
Debugging Information
No technical Problem. So no Debugging
Anything else?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

There is no real answer to solve this problem. The only solution we got is: Don't restrict internet access for at least one endpoint in your environment.

Current solutions:

Don't restrict internet access on one host and pull the images via that host
Use a pull-through cache on a restricted host and unblock the domains as they change, accepting the risk that they can change at anytime
Use a paid distribution which is sustainably funded by the contract you enter
Participating in the project's hosting needs in such a way that we can afford to commit to a particular backing host and have the funding and staffing to delay and communicate any changes (remember the entire internet will depend it for free...)

The reasons for this are outlined clearly in the README and previous discussions: We need flexibility on hosting and last time users got the impression that we'd use a specific backing host and not change it that severely limited our ability to keep hosting sustainable.
And again, we're not preventing you from pulling from a restricted endpoint into your mirror based on the hostnames/IPs that you observed but you take on the risk that we may change the backing hosts at any time. Images already mirrored in your pull-through mirror will be available and you'll just have to update the list of backends you permit.
We're not going to document the endpoints, because then people get the impression that they can make demands about how and when that list is updated, which is not something we have the resources to do.
Please understand that the credits and funding that SIG K8s Infra stewards are responsible for hosting the entire project including CI, scale testing, automation, release pipelines, binary and image hosting, etc. for the 100s of repos under the Kubernetes organization.
We nearly ran out of resources on the GCP account that was funding ~all of this (except a little on other vendors at the time and a lot on legacy Google-internal accounts that can only be accessed by Googlers and had a separate budget) because of excessive spend on image hosting which was consuming more than 2/3 of the total spend and way in excess of the $3M annual public GCP budget (in addition to infra on those internal accounts which e.g. hosted binary releases), which was kept online only by injecting another $700,000 while cutting spend on CI and other project-benefitting infrastructure.
We cannot afford to create commitments that are not resourced.
Please note that there is also no SLA with regards to uptime because this is a free service people work on part time, including people who have other primary work or are entirely volunteer, so if you have strict availability guarantees you need to mirror or use a paid distribution regardless of network restrictions.
We're also subject to the network restrictions of the hosts available to us (ref #138 and other issues) and again without alternate resources there's nothing we can do about it. I highly recommend mirroring or using a distribution with a paid SLA for serious production usage.
Unfortunately you'll either have to deal with the limitations and choose a mirroring approach that works for you between the trade-offs in how you choose to depend on the stability of our hosting, unless your organization can make a sufficient long term funding + staffing commitment to give us room to make more unbounded commitments to users.


chore: getNetworkVersion

:tada: This PR is included in version 9.0.0-beta.6 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 10.0.0-alpha.2 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 10.0.0-beta.1 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 10.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fixing wording.

Hi @AndrejsAnt, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (anant). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

@AndrejsAnt : Thanks for your contribution to the Azure documentation! The author, @stevestein, has been notified to review your proposed change.

@AndrejsAnt - Andrey, you should use the private repo for updates.
@tysonn - this PR is ready to merge.
#sign-off

@tysonn please merge this PR


Operations grid should show more details for BORO ID

Describe the task
The operation grid has a column for BORO ID. If an ID is assigned, its shown, otherwise "N/A" is shown. "N/A" isn't awlays accurate, so instead, we should show:

"Pending for any operation that's not an EIO
"N/A" for EIOs (they'll never get BORO IDs)

Acceptance Criteria

[ ] add conditions to bciers/apps/administration/app/components/datagrid/models/operationColumns.ts
[ ] second
[ ] third

Additional context

Add any other context about the task here.
Or here

@zoeyli-46 , did you just want "N/A" replaced everywhere with "Pending"? "N/A" still makes sense to me for EIOs. Should anything else show based on any other conditions?

N/A for EIOs is perfect, thanks Brianna!!

Hey @BCerki sorry for not catching your question on this earlier. BORO ID's are only assigned to Regulated Operations - so this includes OBPS Regulated Operations, New Entrants and Opt-ins.
Those who do NOT get a BORO ID include all the non-regulated operations - so this includes Reporting Operations, Potential Reporting Operations and EIOs. Let me know if you have any questions! Thanks.

Hey @BCerki sorry for not catching your question on this earlier. BORO ID's are only assigned to Regulated Operations - so this includes OBPS Regulated Operations, New Entrants and Opt-ins. Those who do NOT get a BORO ID include all the non-regulated operations - so this includes Reporting Operations, Potential Reporting Operations and EIOs. Let me know if you have any questions! Thanks.

OK, thanks! I'm going to make a task ticket to update the "Issue BORO ID" button. That ticket only talked about EIOs, so the way it's implemented allows issusing a BORO ID to Reporting Operations, Potential Reporting Operations as well. Ticket: https://github.com/orgs/bcgov/projects/123/views/16?pane=issue&itemId=86401827&issue=bcgov|cas-registration|2453

Thanks @BCerki !


Potential conflict with Inactive Tokens

This warning is logged when moving any token

Are you experiencing any actual conflict between the modules?

Sorry for the late response. Are you still experiencing that issue? If so, does the issue go away if you disable either Drag Ruler or Inactive Tokens?

Sorry for the very late response, I forgot about this until recently testing v11. I don't notice any actual conflict between the modules. The warning does pop up as an overlay on screen every time I move a token for the first time on load so it would be nice if there was a way to fix this but not urgent at all.

Libwrapper is a bit overzealous when showing warnings to users. Getting such a warning is no indication that there are two modules that mod the same portion of foundry (dragging tokens in this case). In that case it shows a warning that there may or may not be a conflict (which is the warning you're getting). You can disable those warnings in the libwrapper settings by disabling "Notify GM of issues". However this has the downside that if you disable that checkbox you won't be notified of actual issues anymore either.


[NEW]

Name: Security fest
Website: www.securityfest.com
Date(s): May 25-26 2023
Location: Gothenburg, Sweden
Twitter/Mastodon Handle: @securityfest
Free: N

Added, thanks for submitting!


Warn ImportApprovedMovie Couldn't import movie

I am running SABnzbd, radarr-sma, and sonarr-sma all in docker containers.  Sonarr seems to be working fine, but radarr-sma does not seem to be executing.  Radarr makes the call to SABnzbd and the movie downloads, but I think it is failing to import and then does not call sma.  Testing the custom script from the radarr UI works fine.  I am using the same settings as in sonarr except calling the postradarr script instead.
From bash within the container, I can get to the config, downloads, and movies folders and see files so I don't think it a volume mount or permissions issue.  Not sure what to try next.
Radarr logs
radarr.4.txt
Radarr-sma logs
radarr-sma.log

Reposted in sickbeard_mp4_automator.
https://github.com/mdhiggins/sickbeard_mp4_automator/issues/1082


Add bulk action to reset currently_in_translation status

Motivation

From time to time, some regions export all pages as XLIFF to get an offer from a translation office, but then don't want to accept it. In these cases, it would be very cumbersome to click through the entire page tree to abort the translation process for every single page.
Proposed Solution

Add a bulk action to abort the translation process for multiple pages at once.
To avoid confusion with the existing bulk actions and to reduce complexity, I would only show this bulk action if at least one page of the current language is currently in translation.
Alternatives

At the moment, I do this with a simple DB query on the production system:
PageTranslation.objects.filter(language__slug="language_slug", page__region__slug="region_slug").update(currently_in_translation=False)

We could also convert this to a management command to allow this task e.g. via:
$ integreat-cms-cli cancel-translation-process <region_slug> <language_slug>

However, this would still require a server admin to manually perform the task, so I would prefer the bulk action.

Isn't such important any more after we do not export xlifss so often. (after MT was implemented). If we have resources, we can implement that but no urgency.

I'm nearly done with this ticket so would like to finish it in the next days


Set minimum width and height for views

Summary
This can help when creating custom ViewGroups (in native modules).
For example: images need to have size in order to be shown, and if the minimum size (width\height) is not set their size is 0.
Changelog
[Android] [Added] - add setMinWidth and setMinHeight to BaseViewManager.
[CATEGORY] [TYPE] - Message
Test Plan

@M-i-k-e-l  Shouldn't we drop in from ViewManager then?

@osdnk What do you mean by "drop in from ViewManager"? Do you mean to move the code to the ViewManager? Because I think the BaseViewManager is more suitable; the ViewManager looks more like an interface.

Hello @M-i-k-e-l, thank you for the contribution. Could you please provide a test plan with an example when this might be useful? As far as I can see we already have minWidth and minHeight properties supported.

Frankly speaking, I don't understand this PR. Can you add some text / gifs / screenshots / snack?


How to change The position of the pageController

I tried to change the position of the pageController ( Dots ) to left alignment. But they are not moving to left. they still reside in the centre position. Any solution  for this.?

Page indicator positioning improvements have been merged to master in #251.


Add activeTextColor to index.d.ts

Overview
There's no type information for activeTextColor property which was added recently in https://github.com/react-native-community/segmented-control/pull/50

Test Plan
I don't really have a test plan for this fix. For now it simply works.

@hardworker released v1.2.1 with the change :+1:


Crypto Wallet and Tracker

An extension for managing cryptocurrency:
Secure wallet storage.
Real-time price tracking.
Transaction history and analytics.
Integration with crypto exchanges.

@taneeshaa15 how many points you earned from my repository. You should have raised 1-2 issues per day, now it looks like spamming

Ma'am I didnot raised any issue yesterday and today also only one issue is assigned to me.
Also, I got 115 points from your repo.

Okay sure.


Add initial README.md for node-problem-detector.

Fix #4.
This PR updated README.md of node-problem-detector.
@dchen1107
/cc @kubernetes/sig-node

@dchen1107 Addressed comments.

LGTM


Copyedits to the video classification example

@fchollet as per your suggestion, I incorporated most of the copyedits. I could not incorporate all of them because the current text's been changed to accommodate the recent changes in our data processing steps. I have also added a further step for the readers to try out Transformer-based models. Let me know your thoughts.

@fchollet I am trying to extend this example to include a minimal Transformers-based model following your book chapter.
Here's how I have defined the model:
num_heads = 1
dense_dim = 4

inputs = keras.Input(shape=(MAX_SEQ_LENGTH, NUM_FEATURES))
x = PositionalEmbedding()(inputs)
x = TransformerEncoder(NUM_FEATURES, dense_dim, num_heads)(x)
x = keras.layers.GlobalMaxPooling1D()(x)
x = keras.layers.Dropout(0.5)(x)
outputs = keras.layers.Dense(5, activation="softmax")(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.summary()

TransformerEncoder is identical to what's shown in the book chapter except I just changed relu to gelu. PositionalEmbedding is defined as follows:
class PositionalEmbedding(keras.layers.Layer):
    def __init__(self, sequence_length=MAX_SEQ_LENGTH, output_dim=NUM_FEATURES, **kwargs):
        super().__init__(**kwargs)
        self.position_embeddings = keras.layers.Embedding(
            input_dim=sequence_length, output_dim=output_dim)
        self.sequence_length = sequence_length
        self.output_dim = output_dim

    def call(self, inputs):
        # Inputs are of shape `(max_seq_length, num_features)`
        length = tf.shape(inputs)[1]
        positions = tf.range(start=0, limit=length, delta=1)
        embedded_positions = self.position_embeddings(positions)
        return inputs + embedded_positions

    def compute_mask(self, inputs, mask=None):
        return tf.math.not_equal(inputs, 0)

    def get_config(self):
        config = super(PositionalEmbedding, self).get_config()
        config.update({
            "output_dim": self.output_dim,
            "sequence_length": self.sequence_length,
        })
        return config

Now, while instantiating the model I am running into:
ValueError: Dimensions must be equal, but are 20 and 1024 for '{{node transformer_encoder_7/multi_head_attention_7/softmax/add}} = AddV2[T=DT_FLOAT](transformer_encoder_7/multi_head_attention_7/einsum/Einsum, transformer_encoder_7/multi_head_attention_7/softmax/mul)' with input shapes: [?,1,20,20], [?,1,20,1024].

The error is of course self-explanatory but I am unable to figure out what I may have missed out. Could you suggest here anything? Here's my notebook.


MGDCTRS-2122 - Ftps/Sftp sink - use enum for File Existence

Failing because of:
https://status.apache.org/incidents/p7d771q6v1fy


update to latest qiskit dev versions

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

7 of 7 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 85.981%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
1104

Relevant Lines:
1284

💛  - Coveralls


How to run pouchd service on system startup?

Why you need it?

When I reboot my host machine, pouchd service doesn't start automatically. (Ubuntu)
How it could be?

Ubuntu 16.04+ uses systemd as system process manager and actually installs one in /usr/lib/systemd/system/pouch.service. However I cannot enable it?
$ sudo systemctl enable pouch
Synchronizing state of pouch.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable pouch
update-rc.d: error: pouch Default-Start contains no runlevels, aborting.

Other related information

$ sudo apt-cache policy pouch
pouch:
  Installed: 1.3.0-0ubuntu16
  Candidate: 1.3.0-0ubuntu16
  Version table:
 *** 1.3.0-0ubuntu16 500
        500 http://mirrors.aliyun.com/opsx/pouch/linux/debian pouch/stable amd64 Packages
        100 /var/lib/dpkg/status

$ uname -a
Linux ubuntu 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Will update this part. Thanks for reporting.


Enforce PHPUnit 9

Pull Request Test Coverage Report for Build 839651164

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+10.9%) to 83.019%

Totals

Change from base Build 765510335:
10.9%

Covered Lines:
44

Relevant Lines:
53

💛  - Coveralls


Add missing definitions to erb.rbi

No signature.
Test plan

See included automated tests.

We have a policy of testing changes to Sorbet against Stripe's codebase before
merging them. I've kicked off a test run for the current PR.  When the build
finishes, I'll share with you whether or how it failed. Thanks!
Stripe employees can see the build result here:
→ https://go/builds/bui_HOrfgGUfFHT09w


[WIP] chore(actions): remove manual source code versioning

Codecov Report
Base: 90.91% // Head: 90.91% // Decreases project coverage by -0.00% :warning:

Coverage data is based on head (9b7d549) compared to base (4179fc6).
Patch coverage: 0.00% of modified lines in pull request are covered.

:exclamation: Current head 9b7d549 differs from pull request most recent head 84caa6c. Consider uploading reports for the commit 84caa6c to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1111      +/-   ##
==========================================
- Coverage   90.91%   90.91%   -0.01%
==========================================
  Files         173      172       -1
  Lines        4930     4929       -1
==========================================
- Hits         4482     4481       -1
  Misses        448      448

Flag
Coverage Δ

py3.8-ubuntu-latest-pandas
90.91% <0.00%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

setup.py
0.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Refresh catalog on every invoke (fresh_catalog: true)

Broad discussion here:

https://github.com/meltano/meltano/discussions/6757

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/2907
Originally created by @vischous on 2021-08-27 12:26:32

I want to have Meltano build a new catalog every run when running meltano invoke tap-oracle
A key at the SingerPlugin level probably makes sense, maybe call it fresh_catalog default to false?
More specific than #2850 as #2627 didn't solve what I'm after. What I really want is a way to manage and watch my catalog change over time (#2677 / #2805 ), but this issue will be an incremental improvement over where I"m at today.
Today I delete the catalog and cache key from .meltano/run/tap-name/*

View 5 previous comments from the original issue on GitLab

Picking up this conversation after @edgarrmondragon pointed me to #2856 in Slack conversation.
Whilst providing and explicit catalog cache refresh via CLI (#2856) and via configuration setting (this issue) is a good medium term option, the important short-term enhancement is actually to address the point that @tayloramurphy  pointed out, which is that the fact of a cached catalog file is currently not mentioned in the documentation.
I've had a quick look through the documentation, and the most sensible place to add a note about the cached catalog (and potentially a hint about using a tap reinstall as a workaround for the missing cache clear feature) seems to be the CLI reference documentation. e.g. in the select section.
Running select is how had my first introduction to the catalog cache (using transferwise tap-postgres). And if I had chosen to start reading the documentation in order to find out why my select output does not match my list of source entities, I probably would have ended up reading the select CLI reference documentation.

Here's some other ideas

Log clearly whether a cached catalog is being used or not (Plus the great docs advice from @cwegener )
What if we cached following a TTL (similar to DNS). Default TTL on invoke would be something like 24 hours, and it'd be configurable. TTL would be ignored if the select changes (just like today), but if the select doesn't change we don't bother updating the Catalog for a day. This was one of the more confusing parts to me as I'd run a project the same way for a while and then realize I'm using an old catalog after a few days / weeks of working.

This could be interesting in the context of:

https://github.com/meltano/meltano/issues/5981

cc @aaronsteers

This would be useful to my team during development, as we're changing the imported schema a lot.

Finally figured out why this is such a frustrating use case for me and haven't articulated it.
When you set something like
  - name: tap-name
    inherit_from: tap-postgres
    select:
    - thissupertable.*

When the thissupertable table gets new columns added the catalog never gets updated because Meltano says "the select statement hasn't changed" therefore everything is good to go.
Whenever I use * in select I implicitly expect Meltano is going to check and update my catalog every run. When I don't use * then I don't' expect it.

Good discussion on this in https://meltano.slack.com/archives/CKHP6G5V4/p1663205560565299
As I think more on this I think having some sort of mechanism to alert on catalog changes would be very beneficial. I'm in favor of a more near-term fix where we can enable users to specify something like refresh_catalog: true but longer term I want to be more thoughtful on the workflows and what we do with the catalog. There's huge value in this metadata to teams and a lot we can do with it (also thinking for managed).

@tayloramurphy and @visch - from the discussion...

https://github.com/meltano/meltano/discussions/6757#discussioncomment-3650870

There seems to be a path forward with catalog_caching being able to be declared as an extra and being able to be disabled with meltano config tap-something set _catalog_caching disabled. One nice thing about not starting with a simple true/false, is that we could expand this in future to be meltano config tap-something set _catalog_caching '60 min' to allow short-lived catalogs in the future.
As an initial boolean toggle though, I think we probably would want to use the true/false or enabled/disabled value to drive the following behaviors:

If caching enabled, use the same catalog file path we have today. (No change.)
If caching is disabled, use a catalog path specific to the execution. (Will not be found on a subsequent run, is multi-process safe, and will still be available afterward for debugging.)

What do you think?

@aaronsteers that seems reasonable as a short-term fix. Long term I like the idea of meltano catalog - I think we can drive huge value around helping with the catalog and alerting on diffs prior to run execution. This is basically the "data contract" of the Singer world...

Arguably done by #8580. Feel free to comment if something's missing.

Arguably done by #8580. Feel free to comment if something's missing.

The PR looks like it solves the problem to me, we'll implement this and come back if it doesnt' work! Thank you, this cleans up a number of things for us

Yeah, just so folks don't have to dig through the PR/docs the options in Meltano 3.5.0a1+ are:

Set the use_cached_catalog extra setting
Use the --refresh-catalog of meltano [run|el|elt|select]


Update CroppedImage PropTypes

Fixes #2647
Changelog Entry
Fixes #2647. Update the CroppedImage component PropType, by @tdurnford in PR #27XX
Description
Updated CroppedImage.propType.height to include strings.

[ ] Testing Added

Coverage remained the same at 65.915% when pulling 6341bcf3895d581f33015f23b641aafc9894bfbb on tdurnford:2647 into 7946bde28d2de8d0d79ae3e738d2acc224184db3 on microsoft:master.


Add AppVeyor Jasmine reporter

Uses the AppVeyor Jasmine reporter from the jasmine-reporters project to report test details to AppVeyor.
Example: https://ci.appveyor.com/project/Daniel15/jest-71o27/build/3/tests (this is building my fork of Jest)
Before: "Tests" tab is empty

After: "Tests" tab is populated

Failing tests show the output when clicked:

Current coverage is 90.31% (diff: 100%)

Merging #1602 into master will not change coverage

@@             master      #1602   diff @@
==========================================
  Files            28         28
  Lines          1125       1125
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1016       1016
  Misses          109        109
  Partials          0          0

Powered by Codecov. Last update fea9b68...557b8af

This is gonna be fun when we kill jasmine!

I'd hope that we have support for our own custom reporters when we kill Jasmine - They're pretty useful for integrating unit testing into other systems. I guess we could use Jest's JSON output instead :smile:


Ensure all spo command have --listId, --listTitle and --listUrl options

Noticed some sort of inconsistency for the spo commands. To reference a list, you could use 3 ways: title, id or URL.
Below is a summary of all commands with their missing option(s).

Command
Options to add
Comments

spo contenttype add
--listId, --listUrl

spo contenttype get
--listId, --listUrl

spo contenttype field remove
--listId, --listUrl

spo field add
--listId, --listUrl
Issue already exists #3627

spo field set
--listUrl

spo list get
--url

spo list roleinheritance break
--listUrl

spo list roleinheritance reset
--listUrl

spo list set
--title, --url
--title already exists, should we change this to --newTtitle?

spo list contenttype add
--listUrl

spo list contenttype list
--listUrl

spo list contenttype remove
--listUrl

spo list contenttype default set
--listUrl

spo list label get
--listUrl

spo list sitescript get
--listUrl

spo list view list
--listUrl

spo list view remove
--listUrl

spo list view set
--listUrl

spo list view field add
--listUrl

spo list view field remove
--listUrl

spo list view field set
--listUrl

spo list webhook add
--listUrl

spo list webhook get
--listUrl

spo list webhook list
--listUrl

spo list webhook remove
--listUrl

spo list webhook remove
--listUrl

spo list webhook remove
--listUrl

spo listitem add
--listUrl

spo listitem attachment list
--listUrl

spo listitem get
--listUrl

spo listitem isrecord
--listUrl

spo listitem list
--listUrl

spo listitem remove
--listUrl

spo listitem roleinheritance break
--listUrl

spo listitem roleinheritance reset
--listUrl

spo listitem set
--listUrl

spo listitem record declare
--listUrl

spo listitem record undeclare
--listUrl

Not sure if we should do this all in one go or consider this as an epic issue.

Nice work @milanholemans! It's a lot of issues if we create them all separately, and the work is rather repetitive. I personally think it would benefit us if we just took it in one go.
What about you?

title already exists, should we change this to --newTtitle?

About this one: that would be a breaking change. I think we should make a separate issue for renaming it, so we can take it up for v6

Nice work @milanholemans! It's a lot of issues if we create them all separately, and the work is rather repetitive. I personally think it would benefit us if we just took it in one go. What about you?

That was exactly what I was thinking. What about your issue? Should we close it then?
However for spo list set where --title is already an existing property, maybe we should remove it from this issue and create a separate issue for the v6 branch?

title already exists, should we change this to --newTtitle?

About this one: that would be a breaking change. I think we should make a separate issue for renaming it, so we can take it up for v6

Great minds think alike 😄

Issue already exists #3627

Yeah, I noticed it too :)
Let's close that one as a duplicate if we agree on doing this in one issue

Even though the work is repetitive, it's a lot of work because it includes adding options, updating command logic for retrieving list, updating docs, adding examples, adding tests, so I'd be all for breaking it up and changing this to an epic.

Even though the work is repetitive, it's a lot of work because it includes adding options, updating command logic for retrieving list, updating docs, adding examples, adding tests, so I'd be all for breaking it up and changing this to an epic.

Would you create 1 issue each command? Will be quite some issues.

What if we'd group similar commands and create an issue for each group?

All done, great job everyone!


Train 251

🚂 :two: :five: :one: 🚋
https://github.com/dcos/dcos/pull/2073 openssl: Disable TLS compression

The TLS compression is vulnerable to MITM attack known as CRIME. This PR disables support for TLS compression in openssl library. None of the components using the openssl will be able to run TLS compressed connection.

https://github.com/dcos/dcos/pull/2062 Remove sensitive config values from diagnostics bundles and build output
Related Issues

https://jira.mesosphere.com/browse/DCOS-19390
https://jira.mesosphere.com/browse/DCOS_OSS-1795

@mesosphere-mergebot bump-ee

@mesosphere-mergebot bump-ee

@mesosphere-mergebot label ready for review

@mesosphere-mergebot merge-it


Write translated text into template and interpolate in js

Alternative approach, writing out the translated strings that will be used in the template.
Possible issues around interpolation - in the example, the 'power' value needs to be written out dynamically from the javascript, but that requires interpolation into the translated string.
The interpolation in JS uses '${}' fields, where the Ruby interpolation uses '%{}', so the delimiters have to be substituted.
The simplistic Javascript interpolation works for basic fields, but wouldn't handle the full translation feature set (e.g. zero/one/many counts etc). That would require a callback to the server.
This example is a bit artificial - probably a better way to achieve translation here would be to simply format the text string on the server and send it with the rest of the results of the Ajax call..


Build failed because of Semantic Issue

I can't build successfully after updating podfile.
Semantic Issue under Realm Framework

Realm framework version: 2.10.2
Xcode version: 8.3.3

Please follow the instructions here and then get back to us if they don't fix your problem.


typo in  RAYLIB_VERSION variable

@AKDev21 Actually this template is designed to work with latest raylib version from GitHub master branch, 3.8.0-dev at this moment, waiting for a new release.

alright


Refactor plugins to use the new routing API

[ ] @backstage/plugin-circleci
[ ] @backstage/plugin-explore
[ ] @backstage/plugin-lighthouse
[ ] @backstage/plugin-sentry
[ ] @backstage/plugin-tech-radar
[ ] @backstage/plugin-techdocs
[ ] ... other plugins

I can help with the @backstage/plugin-api-docs. Let me know if I should prepare a PR.
The only thing that is unclear is how to best distinguish the two routers (root and catalog) since the plugin both can be integrated in the root app and in the service catalog, similar to the techdocs plugin.

@dhenneke that would be super helpful! So from the perspective of plugin in your case it seems like you need to export 2 routers - one for standalone functionality, another one to be a part of catalog's entity page.

So the plugin.ts will setup the router for the plugin and the Router.tsx will setup the router for the catalog integration? I hope that will not lead to confusions in the future. But I will start with that.

@dhenneke it's a bit confusing atm, you are correct here, we are in the middle of propagating the changes everywhere. But the idea is that we go away from registering routes inside plugin.ts and do it directly inside App.tsx, so from that perspective mounting the Router under catalog or on top level of the App makes a very little difference ;)

Okay. Unless you want different routes for different use cases :-). But that's open for future improvements.

Agree, that's what I meant by exporting 2 different routers, one per use case 😀

I've updated the rollbar plugin in #2301

Gonna look at migrating the tech-radar.

I'm taking a look at lighthouse

Taking a look at Jenkins

Jenkins WIP: https://github.com/spotify/backstage/pull/2352

Can these plugins be ran independently now (yarn start from the plugin folder)? When I try it, I see the '404' page and I can't navigate to the plugin. Will local development of plugins that way still be possible? I'm also asking, because I don't understand how entity object could be read if it's not passed as a prop from the catalog router. Or maybe the only way for the local development is to mock it?

@mcalus3 Yep, looks broken right now. We do still want to support isolated plugin development, gonna look for a fix!
Regarding entities the way forward for local development will most likely be to have to mock the entities. It's possible we could provide some base entities to extend as a part of dev-utils though, but nothing decided there yet.


"Method not allowed" when calling OPTION in CORS preflight

After c07f9cbcbc44ce1ef820fc9d4e17b44424f13d7a, OPTIONS doesn't work anymore.

Released in v0.12.7


simplified if-return-return statements

1. Why is this change necessary?
there are some
if ($foo == $bar) { return true } return false;
statements in the code which can be simplified by using only one return statement
2. What does this change do, exactly?
simplifies the statements to
return $foo == $bar
3. Describe each step to reproduce the issue or behaviour.
4. Please link to the relevant issues (if any).
5. Which documentation changes (if any) need to be made because of this PR?
6. Checklist

[ ] I have written tests and verified that they fail without my change
[x] I have squashed any insignificant commits
[x] This change has comments for package types, values, functions, and non-obvious lines of code
[x] I have read the contribution requirements and fulfil them.

Hey @uehler,
we agree with @janbuecker and therefore we will close this PR.


PENV-580: fix metrics duplication

- What I did
fix metrics duplication
- How I did it
- How to verify it
- Description for the changelog

Codecov Report

Merging #162 into master will not change coverage.
The diff coverage is 0.00%.

@@           Coverage Diff           @@
##           master     #162   +/-   ##
=======================================
  Coverage   72.71%   72.71%
=======================================
  Files          55       55
  Lines        3841     3841
=======================================
  Hits         2793     2793
  Misses        894      894
  Partials      154      154

Impacted Files
Coverage Δ

etl/controller/metrics.go
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de1a219...60bb6c7. Read the comment docs.


Code Helper 300% CPU Usage on Mac OS (10.14.1 )

Issue Type: Performance Issue
Extra excludes
"files.watcherExclude": {
"/.DS_Store/": true,
"/.nuxt/": true,
},
"files.exclude": {
"/.nuxt": true,
"/node_modules": true
}
VS Code version: Code 1.34.0 (a622c65b2c713c890fcf4fbf07cf34049d5fe758, 2019-05-15T21:59:22.738Z)
OS version: Darwin x64 18.2.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz (8 x 2700)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: enabledrasterization: enabledsurface_synchronization: enabled_onvideo_decode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
3, 2, 2

Memory (System)
16.00GB (0.17GB free)

Process Argv

Screen Reader
no

VM
0%

Workspace Info
|  Window (vin.js — repo)
|    Folder (repo): 416 files
|      File types: js(161) woff(48) ttf(48) eot(48) vue(19) json(14) env(12)
|                  DS_Store(10) md(10) html(8)
|      Conf files: package.json(6) webpack.config.js(4);

Extensions (8)

Extension
Author (truncated)
Version

gitlens
eam
9.8.0

vscode-angular-html
gha
1.1.1

vscode-angular-components-intellisense
ipa
1.0.0

node-module-intellisense
lei
1.5.0

vscode-language-babel
mgm
0.0.22

dotenv
mik
1.0.1

vetur
oct
0.21.0

webstorm-icons
pms
1.0.0

(4 theme extensions excluded)

As a temp solution, you can decrease your VS Code typescript version to 1.3.3333. It was helped me. Other proposes had zero results.
Here you can find how to specify typescript version.
https://stackoverflow.com/questions/39668731/what-typescript-version-is-visual-studio-code-using-how-to-update-it

As a temp solution, you can decrease your VS Code typescript version to 1.3.3333. It was helped me. Other proposes had zero results.
Here you can find how to specify typescript version.
https://stackoverflow.com/questions/39668731/what-typescript-version-is-visual-studio-code-using-how-to-update-it

Please follow these instructions to collect performance information. Share the output of code --status

Please follow these instructions to collect performance information. Share the output of code --status

As a temp solution, you can decrease your VS Code typescript version to 1.3.3333. It was helped me. Other proposes had zero results.

I downgraded VS Code to 1.33.1 and It helped.

That trace shows no cpu usage from code. Are you sure you captured this when the high cpu usage was happening?

I have the same issue. I've disabled all the extensions and added node_modules et al to watcherExclude. I stopped using Atom for this same issue.
If I stop using VSCode for a bit, it calms... as soon as I start working again, cpu goes nuts.
This only happens with React web apps so far.. React Native nope...
Version:          Code 1.34.0 (a622c65b2c713c890fcf4fbf07cf34049d5fe758, 2019-05-15T21:59:22.738Z)
OS Version:       Darwin x64 18.6.0
CPUs:             Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz (8 x 2800)
Memory (System):  16.00GB (0.95GB free)
Load (avg):       3, 3, 3
VM:               0%
Screen Reader:    no
Process Argv:     .
GPU Status:       2d_canvas:                    enabled
                  checker_imaging:              disabled_off
                  flash_3d:                     enabled
                  flash_stage3d:                enabled
                  flash_stage3d_baseline:       enabled
                  gpu_compositing:              enabled
                  multiple_raster_threads:      enabled_on
                  native_gpu_memory_buffers:    enabled
                  rasterization:                unavailable_off
                  surface_synchronization:      enabled_on
                  video_decode:                 enabled
                  webgl:                        enabled
                  webgl2:                       enabled

CPU %	Mem MB	   PID	Process
    0	    98	  9455	code main
    0	    49	  9518	   gpu-process
    0	    82	  9547	   shared-process
    0	   262	  9923	   window (user.js — [app])
    1	    82	  9924	     extensionHost
  108	   442	  9927	       electron_node tsserver.js
    0	    49	  9943	         electron_node typingsInstaller.js typesMap.js
    0	    33	  9942	       /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /Applications/Visual Studio Code.app/Contents/Resources/app/extensions/json-language-features/server/dist/jsonServerMain --node-ipc --clientProcessId=9924
    0	    49	 10115	       electron_node server.js
    0	    33	  9925	     watcherService

Workspace Stats:
|  Window (user.js — [app])
|    Folder ([app]): 51 files
|      File types: js(27) json(5) map(4) DS_Store(3) ico(2) html(2) css(2)
|                  md(1) svg(1) ttf(1)
|      Conf files: package.json(1)

settings:
"files.exclude": {
        "**/.git": true,
        "**/.svn": true,
        "**/.hg": true,
        "**/.DS_Store": true,
        "**/bower_components": true,
        "**/jspm_packages": true,
        "**/node_modules": true
    },
    "search.exclude": {
        "/node_modules": true
    },
    "files.watcherExclude": {
        "**/.git": true,
        "**/.svn": true,
        "**/.hg": true,
        "**/.DS_Store": true,
        "**/bower_components": true,
        "**/jspm_packages": true,
        "**/node_modules": true
    }

@SirCameron Please open a separate issues, but also make sure you use the latest VS Code insiders build

@mjbvz it's the same issue, though?

@SirCameron Open a separate issue and test using the latest VS Code insiders build


Servo Plus - Setting angle to zero causes a delay in movement

This issue was caused by the fault of the servo, after using a different servo it worked flawlessly.


Improve 'known issue' feature

Currently we analyze a big part of issue's stack trace, so, any updates cause that zafira not able to determine known issues. Is it any way to improve this flow?

Let's assume that we have some general flow of registration or log in and there is scope of known issues in the app.  One day devs added additional screen during registration or log in (for example confirm privacy or enable notifications).  As a result all known issues will be lost.

It would be a good point and opportunity to have ability exclude from a test log all dynamic capabilities with alphanumeric dynamic part that can influence to a logs comparison process.
e.g. userDataDir: /tmp/.com.google.Chrome.IbEp4t

moved to another repo


Preparation for the CPI Workshop Presentation

Describe the task
This ticket captures work needed for preparing to present the presentation between Suha & Nan
Acceptance Criteria

[x] Reach out to Patricia and Jaimie for feedback on Menti questions
[x] Update Menti questions
[x] Split presentation work
[x] Do a dry-run of the presentation on the 3rd of May

Additional context

Add any other context about the task here.
Or here

@nanyangpro @Dianadec created this ticket for upcoming planning for the CPI workshop

Thanks @suhafa! A test run of the CPI workshop has been schedule on next Wed, May 3 for the whole branch. Shall we have a prep meeting together next Tue, May 2 to go through the deck and plan together for our team's session?

As outcome of the Test Run (Wed, May 3):

CPI Workshop May 2023 presentation deck has been refined with the updated version sent to Alana C. to be incorporated in the master slide deck for the workshop

Hi @pbastia! This task is completed and ready for your review:

We reached out to program area and refined engagement questions on Menti
Nan & I split the work for the presentation
We completed the dry-run prep as well

Thank you! Fantastic job at the presentation :)


[V3] Displacement Filter inside mask deform mask

Hi!
I'm currently trying to make a mask (with the new V3 masking method! :) ) on a sprite and then add a Displacement filter to this sprite,
the behavior I expect is that only the sprite should be deformed, inside the rounded mask,
the result is that the mask is deformed too..
see : http://codepen.io/jeremypetrequin/pen/vEwbdB?editors=001
I've tried with several calls to render or change order of instructions, but I can't have what I want..
Does someone know if it's possible?
(I know I could make this simple mask with a PIXI.Graphics but my real mask is way more complicated than that ;) )
And two others small things:

I think there's maybe a typo in https://github.com/GoodBoyDigital/pixi.js/blob/dev/src/filters/displacement/DisplacementFilter.js#L13
the constructor needs a PIXI.Sprite and the comment says Texture
Is there somewhere to get some start of documentation and/or examples for the V3? Or do you have an idea about a release date maybe? :)

Thanks everybody!

Hi there! the way v3 works is that it always applies the filters as the last step of rendering an object.
If you nest your displacement object in masked container it should get you the effect you require:
http://codepen.io/anon/pen/ogrPyo
I also tweaked the documentation in the code to. The latest docs can be found here: http://pixijs.github.io/docs/
Cheers!


Feature Request: Contributing.md ?

I like that you note at the bottom of this that others should contribute. Perhaps we can expand upon that a little further, and add a Contributing.md file with instructions?
I'm curious about whether contributions are preferred in a certain format, say forking the repo, then making a change, and submitting it as a pull request? Or is it more open than that?
More importantly, what kind of contributions are welcome or needed? Are there areas that have been identified as needing further development? I don't necessarily agree with all of the items here, and I have come across my own books and courses that I think are better suited, but is this purely a big list that we add to, and not replace or substitute existing items?
Thanks for putting this repo together, it is very valuable and I refer to it often :)

+1 to a "How to Contribute" section.
One of the things I am finding lacking is some kind of estimate for each task/course/book on the list. For example the Statistics I Princeton / Coursera class, is going to be a 12 week class - when offered next - and they suggest you will need 5-8 hours/week to complete the course. I'd love to look through all of classes/books/etc and surface some of that information directly into the list. Would this be a welcome pull request?
Similar, but slightly different, It might be helpful to spell out "How to take this Masters" as well. Should my goal be to be able to say "I completed The Open Source Data Science Masters" or should my goal be to say "I created and completed my own Open Source Data Science Masters"? I think I prefer the former as attaching to a group lends credibility, even if that group is an open source community-based curriculum. This leads me to ask about the role of the /transcripts folder - is that something open to any and all, or would I manage my transcript in my own fork of this repo?

Much of the qualitative feedback offered in the past has been taken into account in the rebuilt OSDSM, pushed late last year. Please take a look and relay any updated feedback! Thank you for your thoughtful suggestions about how to engage folks here.


[GOBBLIN-1741] Create manifest based dataset finder

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[ ] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-1741

Description

[ ] Here are some details about my PR, including screenshots (if applicable):
Create manifest based dataset finder

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:
unit test, test on azkaban to see it can copy data

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #3598 (37c0c9f) into master (c3d1ba8) will decrease coverage by 2.99%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3598      +/-   ##
============================================
- Coverage     46.89%   43.89%   -3.00%
+ Complexity    10673     2063    -8610
============================================
  Files          2120      408    -1712
  Lines         83080    17618   -65462
  Branches       9254     2153    -7101
============================================
- Hits          38962     7734   -31228
+ Misses        40548     9027   -31521
+ Partials       3570      857    -2713

Impacted Files
Coverage Δ

...a/org/apache/gobblin/cluster/GobblinHelixTask.java
81.72% <0.00%> (-2.16%)
:arrow_down:

...che/gobblin/data/management/copy/CopyableFile.java

...che/gobblin/metrics/kafka/KafkaSchemaRegistry.java

...lin/eventhub/writer/BatchedEventhubDataWriter.java

...e/gobblin/converter/SchemaConversionException.java

...pache/gobblin/yarn/YarnContainerRequestBundle.java

.../apache/gobblin/http/ApacheHttpResponseStatus.java

.../service/monitoring/KafkaAvroJobStatusMonitor.java

...e/gobblin/yarn/event/ContainerShutdownRequest.java

...in/azkaban/AzkabanGobblinLocalYarnAppLauncher.java

... and 1705 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[release-1.3] fix indention issue in egressgw env.

This is an automated cherry-pick of #340

@googlebot I consent.


fix artifact name collisions

cf https://github.com/actions/upload-artifact/issues/481#issuecomment-1862899981


etcdserver: range queries with min/max create revision

More API updates pursuant to #5412 / reducing sychronization message overhead.

LGTM

@heyitsanthony Shall we get this merged?


Latest tagged version produces an error when Go module mode is enabled

I attempted to import "github.com/zoom-lib-golang/zoom-lib-golang"
As demonstrated at: https://play.golang.org/p/U79iwt7vaAh
It fails with:
go: finding module for package github.com/zoom-lib-golang/zoom-lib-golang
go: downloading github.com/zoom-lib-golang/zoom-lib-golang v1.0.0
go<EMAIL_ADDRESS>parsing go.mod:
	module declares its path as: github.com/himalayan-institute/zoom-lib-golang
	        but was required as: github.com/zoom-lib-golang/zoom-lib-golang

The import fails when Go module mode is enabled because the latest tagged version of the module has a different address than the current address: v1.0.0.
A solution would be to add a new tag with the address updated in the go.mod file.

Published a new version, please check back in ~1hr to see if this is fixed

There seems to be something cached on the golang playground, can you confirm if this now works correctly locally? I expect that it should.

I believe you added the v2.0.0 tag. When you tag a package with a different major version you've got to change the import path to reflect that. I think tagging TIP with v1.0.1 would work without having the v2.0.0 tag.

I'm sorry, but I don't quite understand what you're saying or why it's not working on Go playground. I tagged it v2.0.0 because there are backwards incompatible changes from v1.0.0, so v1.0.1 would be semantically inaccurate. Additionally, the module path in the go.mod file refers to the updated package path.
I tried also adding a retract v1.0.0 line to the current go.mod file. Not really sure what else to do here besides tagging it v1.0.1, which seems incorrect.

I updated the urls in the v1.0.0 tag and force updated the tag, and I also tagged that commit as v1.0.1, hopefully one of these things will do the trick.

Well now it's saying I need to run go mod tidy... that's progress, except that when I run that locally on any of the tags or main, it does not change anything

@rafecolton can you publish a new version? v1.0.2
When you force publish versions like this the go checksum db is violated and downstream consumers have issues like so:
<EMAIL_ADDRESS>verifying go.mod: checksum mismatch
	downloaded: h1:t3p44iNBETLiJzk0HTH42PumtcP3AHi+Pd/ZY0SPpng=
	sum.golang.org: h1:yO9Ob+in8Nn1EnyCoEPoCwD20L+5zD9OIgODUb4VdXM=

Bump

Just pushed a tag for v1.0.2 - please let me know if that resolves your issue (or reopen this ticket if it does not).

Thank you @rafecolton!


WASM+Identity same-site & antiforgery updates

Fixes #31389
Addresses #31205
Addresses #28161
Sample updates
FIRST ... before reviewing the article DIFF on this PR ... let's sort out the sample updates at ...
https://github.com/dotnet/blazor-samples/pull/210

NIT updates on the PR

Move CORS in the pipeline above the logout endpoint.
Lowercase the logout endpoint from /Logout to /logout.
Change conditional check empty != null to empty is not null.
Place a line of spacing above returns ... that's by convention for Blazor example code.
Remove null-forgiving ops from email and password, as they can't be null.
Remove the UserBasic class, as it isn't used.

Same-site/secure policy settings: Add commented-out default-setting code for it.
Add a quick-'n-dirty data processing experience.

Form in a page that uses the Auth client to POST to the Backend app.
Minimal API endpoint that processes the data and returns a string.
~NOTE: The same errors shown in https://github.com/dotnet/blazor-samples/issues/161 are seen here when that code processes an unauthenticated request to the /data-processing endpoint. I'll place the error at the bottom of this OP.~ We think now that this is resolved by the sample updates.

Article updates
Security Overview
This section applies to all Blazor app hosting models/templates. The primary goal here is to add a reminder that ...

~Antiforgery mitigation is for form submissions encoded as application/x-www-form-urlencoded, multipart/form-data, or text/plain with a cross-link to the Blazor Forms article coverage on forms antiforgery support.~ Updated on review.
~Server API endpoints with application/json-encoded content and CORS don't require CSRF protection.~ Updated on review.

Standalone with Identity article

Adds a new section on Cross-domain hosting (same-site configuration).
Adds a new section on antiforgery support.

Explain the situation with the logout endpoint.
Add the reminders on forms antiforgery and server API/JSON/CORS.

I'm still a bit concerned ... even if this is all correct (or close to correct) ... that the bit about forms submission in a standalone WASM situation could use more work. Using an HTTP client with a JSON request and CORS is what we focus on throughout the docs. This will be the first time that we explicitly say that antiforgery isn't a concern in that scenario. However, EditForm is useful client-side to just collect the data for a JSON POST to a server API. I don't want readers to think that anything is happening for EditForm in such an app vis-a-vis antiforgery. I might need to add more content on this point.
... and BTW Jeremy ... it seems to me that one day if I ever get the time that I should convert your plain forms over to EditForm forms with the nice validation ... and roll in the bits to trigger validation failures with the ValidationMessage component. I can do all that ... it's just a question of TIME ⏲ ... and I won't have any for the foreseeable future 🏃♂😄.
Error
This has been addressed by the updates to the sample app.
~Error for unauthenticated hit on the /data-processing endpoint. What the sample is doing now is trapping it in a try-catch and showing a message. Can't we work out something more graceful than that? ... or does this all resolve back to your remarks here ...~
~https://github.com/dotnet/blazor-samples/issues/161#issuecomment-1887754626~

Internal previews

📄 File
🔗 Preview link

aspnetcore/blazor/security/index.md
ASP.NET Core Blazor authentication and authorization

aspnetcore/blazor/security/webassembly/standalone-with-identity.md
Secure ASP.NET Core Blazor WebAssembly with ASP.NET Core Identity

Thanks @halter73. I'll wait a day for a response to your question ...

@blowdart @GrabYourPitchforks Do either of you think we should hedge this statement in some way?

I can't wait too long ... I need to unblock things fairly fast these days for more work ⛰⛏😅.


Flexible grammar installer

Instead of hard-coding the grammars and queries I added the possibilities to install them and add them in the init.lua user configuration file.
Now it is possible to configure its own grammar from local files or from remote git repositories (doc and examples in the README.md)

you could've used the existing installer functions to facilitate this feature, no?
i've also realized that this removes defaults from evergreen and users need to find the parsers to use for their languages. it adds extra effort for no good reason.
i would prefer to not remove the ease of use and only add the ability to configure extra parsers (which would be none in the future, optimally) and not all required ones of a user.

Hi @TorchedSammy, thank you for your feedback!
I did not think about Windows users (as I am not used to deal with them ;) ) but your concern makes completely sense!
I have the following proposal:

Camel case function naming does not seem to be the standard as lite-xl-lsp or console use snake case, and these are officially supported plugin (or at least managed by the core team). So I believe this should be the convention (or at least for the functions that are called in the init.lua). For the commits I never heard about conventional and I will look into it. Maybe it can be useful to have some sort of guidelines about it in the project.
I think I could have a solution on how to merge the simplicity and flexibility:
I will add another installation method that will download the pre-compiled library
I will add a structure (like yours) with some default languages and their configuration if the user adds a language without specifying any path/url/git the plugin will try to use the default config (if any)
I will add back the default queries as for the previous point if the user does not specify any queries parameter and an existing default query exists this one will be used
I will add a command to display the status of the plugin (installed grammars, file extension and so on)
I do not think having a command install is a good idea as it hides too much information to the user (as it is finally a hidden configuration) and it makes more difficult to manage the grammars I think that add the default and simply add languages.add_language("c") to the init.lua is easy enough as well as very clear.

Let me know what do you think about my proposal.
Cheers!
Martino

camel case is what i prefer for my code, so it will be what is used here.
all the suggestions here sound good, but let's see how it'll turn out?
i would prefer to keep the install command, no real reason to remove it. though other contributors can comment @xcb-xwii @takase1121

overall :+1:

I put back the queries and added the defaults installer back (look at the README.md for more information) and added the status command to check installed plugins, it shows the result (installed grammars and grammars that have been configured but failed to install) in the logs but I am not sure if it is the best solution.
I did not add back the install command as it will not fit the logic to manage the configurations (as the installation is made automatically at initialization if the library or the query is not found in the data folder).
It could be add in the form of a work around (at initialization you could check for all the defaults languages and check if they are installed in the data folder) but is not very elegant.
I tried to use the conventional standard for the commits.

i would prefer to keep the install command, no real reason to remove it. though other contributors can comment @xcb-xwii @takase1121

I also do not really see a reason to do so, but I do not mind removing it.
It might be useful to be able to install parsers on the fly without having to restart / reload, but it's not too big of a benefit to me.
However, if we introduce ways to update parsers, it might be worth it to have commands to manage parsers, so we can selectively update parsers, and also not have them all update immediately on startup.

There are some problems with the implementation here. This PR, as far as I can tell, does not attempt to check for updates / changes to the Git repo, and for the URL, it just re-downloads every time (do correct me if I'm wrong on these). Additionally, if a parser is already installed in a directory, replacing the parser with a different one inside the same directory requires you to remove the directory first, which is something we do not provide.
Another major issue right now is that we provide no way to update and remove parsers, which I've mentioned above. The more declarative installation methods implemented by this PR are nice, but it needs improvement.

These changes seems to have removed the default Git repositories for the parsers. I think that needs to be restored, since not everyone wants to, or can use our prebuilt binaries, since only binaries for amd64 and select OSes are provided. A config option should also be added to choose between the prebuilt binaries or download from default repos for the built-in parsers.

@xcb-xwii I do agree with you that something to manage parsers (at least to updates and clean the ones no more used) should be implemented (and I was planning to do so).
For the moment at the initialization I just check if for the languages configured exists the parser library and the queries, if not I try to install them using the user configuration. But this is somehow similar to how it is done now (in the sense that evergeen does not check if there is a new version of the installed parsers or anything else, but it just installs the current version whenever the installation command is issued).
Check automatically for updates could slow down the initialization of lite (in particular if many parsers are configured), so I propose to add an update and a clean command to do so.

These changes seems to have removed the default Git repositories for the parsers. I think that needs to be restored, since not everyone wants to, or can use our prebuilt binaries, since only binaries for amd64 and select OSes are provided. A config option should also be added to choose between the prebuilt binaries or download from default repos for the built-in parsers.

Good suggestion

These changes seems to have removed the default Git repositories for the parsers. I think that needs to be restored, since not everyone wants to, or can use our prebuilt binaries, since only binaries for amd64 and select OSes are provided. A config option should also be added to choose between the prebuilt binaries or download from default repos for the built-in parsers.

Good suggestion

@xcb-xwii now there is the possibility to chose to use the precompiled parser or the default git for the "supported" languages (by default, if not specify it will use the precompiled)

@xcb-xwii I do agree with you that something to manage parsers (at least to updates and clean the ones no more used) should be implemented (and I was planning to do so). For the moment at the initialization I just check if for the languages configured exists the parser library and the queries, if not I try to install them using the user configuration. But this is somehow similar to how it is done now (in the sense that evergeen does not check if there is a new version of the installed parsers or anything else, but it just installs the current version whenever the installation command is issued). Check automatically for updates could slow down the initialization of lite (in particular if many parsers are configured), so I propose to add an update and a clean command to do so.

@xcb-xwii, @TorchedSammy I added the following commands:

evergreen:update to update a single parser/queries
evergreen:update-all to update all parsers/queries
evergreen:clean to clean up unused parsers/queries
evergreen:status to check the available grammars

I do not think to implement any other features, as I think is now quite complete and flexible.
Best luck!


Creating my portfolio website

Creating my sample portfolio website.

Why did you make this as a PR (pull request) they mean that you want to incorporate these changes into the repository. So by making this PR your saying that you want every single minimal user to see your personal portfolio when they start the default template.

Also please use https://github.com/BDHU/minimalist to base your portfolio site off its the same as minimal just with a dark theme and bug fixes and it's also maintained


Anvil fork throws an error when receiving large gas limits

Component
Anvil
Have you ensured that all of these are up to date?

[X] Foundry
[X] Foundryup

What version of Foundry are you on?
anvil 0.1.0 (22ad954 2022-10-29T00:06:21.340549Z)
What command(s) is the bug in?
No response
Operating System
macOS (Intel)
Describe the bug
Anvil throws HttpProviderError when a custom gas limit is passed as kwarg to an ethers contract function, that is larger than the block limit of the forked network.
For example:
  const results = await this.pathFinder.callStatic.findAllSwaps(swapTask, {
    gasLimit: 400e6
  });

would fail when run on Anvil forking the mainnet. This is despite a larger block limit being set in foundry.toml.
This breaks functionality involving static calls that consume more than a block of gas - even though node providers allow up to 300-500M of gas per static call. hardhat node in forking mode also allows large static calls, as long as a custom gas limit is provided.

what's the actual error message here?

Error: missing revert data in call exception; Transaction reverted without a reason string [ See: https://links.ethers.org/v5-errors-CALL_EXCEPTION ] (data="0x", transaction={"gasLimit":{"type":"BigNumber","hex":"0x11e1a300"},"to":"0x5BA1e12693Dc8F9c48aAD8770482f4739bEeD696","data":"0x252dba420000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000500000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000022000000000000000000000000000000000000000000000000000000000000002e000000000000000000000000000000000000000000000000000000000000003a00000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000b0e2061b34171edc2b3c34e86b0b1f05ec7eeffc000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000614d98a57a5d879d717152de0690ed2b04562ade000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000cdc8ba9815c76e838a759e932f57bf6f990a6e71000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000bdfa4f4492dd7b7cf211209c4791af8d52bf5c50000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f70000000000000000000000000035fc5208ef989c28d47e552e92b0c507d2b31800000000000000000000000000000000000000000000000000000000","accessList":null}, error={"stack":"Error: processing response error (body="{\"jsonrpc\":\"2.0\",\"id\":79,\"error\":{\"code\":-32603,\"message\":\"EVM error CallerGasLimitMoreThenBlock\"}}", error={"stack":"Error: EVM error CallerGasLimitMoreThenBlock\n    at getResult (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/providers/src.ts/json-rpc-provider.ts:142:28)\n    at processJsonFunc (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/src.ts/index.ts:383:22)\n    at /Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/src.ts/index.ts:320:42\n    at step (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:33:23)\n    at Object.next (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:14:53)\n    at fulfilled (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:5:58)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","message":"EVM error CallerGasLimitMoreThenBlock","code":-32603}, requestBody="{\"method\":\"eth_call\",\"params\":[{\"gas\":\"0x11e1a300\",\"to\":\"0x5ba1e12693dc8f9c48aad8770482f4739beed696\",\"data\":\"0x252dba420000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000500000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000022000000000000000000000000000000000000000000000000000000000000002e000000000000000000000000000000000000000000000000000000000000003a00000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000b0e2061b34171edc2b3c34e86b0b1f05ec7eeffc000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000614d98a57a5d879d717152de0690ed2b04562ade000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000cdc8ba9815c76e838a759e932f57bf6f990a6e71000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000bdfa4f4492dd7b7cf211209c4791af8d52bf5c50000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f70000000000000000000000000035fc5208ef989c28d47e552e92b0c507d2b31800000000000000000000000000000000000000000000000000000000\"},\"0xf273f6\"],\"id\":79,\"jsonrpc\":\"2.0\"}", requestMethod="POST", url="http://localhost:8545", code=SERVER_ERROR, version=web/5.7.1)\n    at Logger.makeError (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/logger/src.ts/index.ts:269:28)\n    at Logger.throwError (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/logger/src.ts/index.ts:281:20)\n    at /Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/src.ts/index.ts:341:28\n    at step (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:33:23)\n    at Object.next (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:14:53)\n    at fulfilled (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:5:58)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","message":"processing response error (body="{\"jsonrpc\":\"2.0\",\"id\":79,\"error\":{\"code\":-32603,\"message\":\"EVM error CallerGasLimitMoreThenBlock\"}}", error={"stack":"Error: EVM error CallerGasLimitMoreThenBlock\n    at getResult (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/providers/src.ts/json-rpc-provider.ts:142:28)\n    at processJsonFunc (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/src.ts/index.ts:383:22)\n    at /Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/src.ts/index.ts:320:42\n    at step (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:33:23)\n    at Object.next (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:14:53)\n    at fulfilled (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:5:58)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","message":"EVM error CallerGasLimitMoreThenBlock","code":-32603}, requestBody="{\"method\":\"eth_call\",\"params\":[{\"gas\":\"0x11e1a300\",\"to\":\"0x5ba1e12693dc8f9c48aad8770482f4739beed696\",\"data\":\"0x252dba420000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000500000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000022000000000000000000000000000000000000000000000000000000000000002e000000000000000000000000000000000000000000000000000000000000003a00000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000b0e2061b34171edc2b3c34e86b0b1f05ec7eeffc000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000614d98a57a5d879d717152de0690ed2b04562ade000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000cdc8ba9815c76e838a759e932f57bf6f990a6e71000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000bdfa4f4492dd7b7cf211209c4791af8d52bf5c50000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f70000000000000000000000000035fc5208ef989c28d47e552e92b0c507d2b31800000000000000000000000000000000000000000000000000000000\"},\"0xf273f6\"],\"id\":79,\"jsonrpc\":\"2.0\"}", requestMethod="POST", url="http://localhost:8545", code=SERVER_ERROR, version=web/5.7.1)","reason":"processing response error","code":"SERVER_ERROR","body":"{"jsonrpc":"2.0","id":79,"error":{"code":-32603,"message":"EVM error CallerGasLimitMoreThenBlock"}}","error":{"stack":"Error: EVM error CallerGasLimitMoreThenBlock\n    at getResult (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/providers/src.ts/json-rpc-provider.ts:142:28)\n    at processJsonFunc (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/src.ts/index.ts:383:22)\n    at /Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/src.ts/index.ts:320:42\n    at step (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:33:23)\n    at Object.next (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:14:53)\n    at fulfilled (/Users/mikael/Coding/gearbox/liquidator-v2/node_modules/@ethersproject/web/lib/index.js:5:58)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","message":"EVM error CallerGasLimitMoreThenBlock","code":-32603},"requestBody":"{"method":"eth_call","params":[{"gas":"0x11e1a300","to":"0x5ba1e12693dc8f9c48aad8770482f4739beed696","data":"0x252dba420000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000500000000000000000000000000000000000000000000000000000000000000a00000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000022000000000000000000000000000000000000000000000000000000000000002e000000000000000000000000000000000000000000000000000000000000003a00000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000b0e2061b34171edc2b3c34e86b0b1f05ec7eeffc000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000614d98a57a5d879d717152de0690ed2b04562ade000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000cdc8ba9815c76e838a759e932f57bf6f990a6e71000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f7000000000000000000000000bdfa4f4492dd7b7cf211209c4791af8d52bf5c50000000000000000000000000000000000000000000000000000000000000000000000000000000000a2ca503153cd5cb2892a0928ac0f71f49a3c194000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000440dbd616d000000000000000000000000672461bfc20dd783444a830ad4c38b345ab6e2f70000000000000000000000000035fc5208ef989c28d47e552e92b0c507d2b31800000000000000000000000000000000000000000000000000000000"},"0xf273f6"],"id":79,"jsonrpc":"2.0"}","requestMethod":"POST","url":"http://localhost:8545"}, code=CALL_EXCEPTION, version=providers/5.7.2)

Closing, fixed by #4324 and a few other gas estimation ports we got from Reth.


Add shuffle support to TVM

Actually, TVM's IR support is incomplete.
Shuffle's IRVisitor dispatch is missing.
It has no LLVM generator either.

@tqchen

@kazum

@ZihengJiang

@zhiics I have resolved most of your concerns and answered your question.

@kazum @tqchen @ZihengJiang Can any of you take a look as well?

@were please also use pattern-match to add most of the support(concat and subsetting) to CodgenC(CUDA/OpenCL/Metal)

@tqchen I added CUDA backend.
@kazum I checked the existence and bound of the extracted IntImm and UIntImm.

@kazum Can you take a look again?
@tqchen Can we merge this for now? I am in a hurry to use Shuffle.


Adds options to foodcritic linter

Adds a way to pass command line options to the foodcritic command and
documentation about it.

Cheers! :beers:


ci: add typo checking

Since this is a public-facing documentation/resource repository,
we should be extra-careful and
judicious with typos.
typos is a powerful source code spell checker.
I've added a CI job that runs on every PR and push to master (but can also be run manually with workflow_dispatch) that checks for typos.
I've also added a config file .typos.toml that deals with false positives and ignore some vendor related files and some filetypes that we don't want to check/correct for typos,
e.g. lock files.
Finally, I've also corrected a few typos that were in the codebase that typos flagged.
If you want to run yourself you can do a cargo install (or binstall) the typos-cli:
cargo install typos-cli

It is also available in several pkg managers,
check the installation options at the typos repo

PS: If you find this useful, I can PR this across the rust-bitcoin crate ecosystem.
Cc @apoelstra.

utACK 81bbee2cb82a54c81dbe7b8f60042c8cbf61b6f7.
I'm a bit confused about how this works and distinguishes between bech32/base64/base58/hex words and "real" words. But in general I think it's a good idea and we should have some sort of automatic typo checking. Would also be happy to have this elsewhere in the rust-bitcoin ecosystem.

nit: There is a newline missing from the workflow. In case its interesting to you I have my editor configured to add a newline at end of file on save if its missing.

Done!

Needs rebase mate.

It looks like now there is a merge commit and an unsigned commit :). Can you try rebasing again.

Still not quite right :)
I pulled it down and had a look, I think you want to hard reset away the last patch and then rebase on master and fix the two conflicts. You should be left with a single patch.

Sorry, I was with a unsynced fork. Took me a while to figure out that this was the issue.
But I think it's ready now