[Bug Report] Unsupported scheduler error when training SAE

If you are submitting a bug report, please fill in the following details and use the tag [bug].
Describe the bug
Example SAE training code encounters an error in sae-lens==0.7.0
Code example
import os

import torch

os.environ["TOKENIZERS_PARALLELISM"] = "false"
os.environ["WANDB__SERVICE_WAIT"] = "300"

from sae_lens import LanguageModelSAERunnerConfig, language_model_sae_runner

cfg = LanguageModelSAERunnerConfig(
    # Data Generating Function (Model + Training Distibuion)
    model_name="gpt2-small",
    hook_point="blocks.2.hook_resid_pre",
    hook_point_layer=2,
    d_in=768,
    dataset_path="Skylion007/openwebtext",
    is_dataset_tokenized=False,
    # SAE Parameters
    expansion_factor=64,
    b_dec_init_method="geometric_median",
    # Training Parameters
    lr=0.0004,
    l1_coefficient=0.00008,
    lr_scheduler_name="constantwithwarmup",
    train_batch_size=4096,
    context_size=128,
    lr_warm_up_steps=5000,
    # Activation Store Parameters
    n_batches_in_buffer=128,
    training_tokens=1_000_000 * 300,
    store_batch_size=32,
    # Dead Neurons and Sparsity
    use_ghost_grads=True,
    feature_sampling_window=1000,
    dead_feature_window=5000,
    dead_feature_threshold=1e-6,
    # WANDB
    log_to_wandb=True,
    wandb_project="gpt2",
    wandb_entity=None,
    wandb_log_frequency=100,
    # Misc
    device="cuda",
    seed=42,
    n_checkpoints=10,
    checkpoint_path="checkpoints",
    dtype=torch.float32,
)

sparse_autoencoder = language_model_sae_runner(cfg)

Error message:
Traceback (most recent call last):
  File "/home/daniel/ml_workspace/SAELens/scripts/train_gated_sae.py", line 51, in <module>
    sparse_autoencoder = language_model_sae_runner(cfg)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/ml_workspace/SAELens/sae_lens/training/lm_runner.py", line 34, in language_model_sae_runner
    sparse_autoencoder = train_sae_on_language_model(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/ml_workspace/SAELens/sae_lens/training/train_sae_on_language_model.py", line 92, in train_sae_on_language_model
    return train_sae_group_on_language_model(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/ml_workspace/SAELens/sae_lens/training/train_sae_on_language_model.py", line 132, in train_sae_group_on_language_model
    train_contexts = {
                     ^
  File "/home/daniel/ml_workspace/SAELens/sae_lens/training/train_sae_on_language_model.py", line 133, in <dictcomp>
    name: _build_train_context(sae, total_training_steps)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/ml_workspace/SAELens/sae_lens/training/train_sae_on_language_model.py", line 302, in _build_train_context
    scheduler = get_scheduler(
                ^^^^^^^^^^^^^^
  File "/home/daniel/ml_workspace/SAELens/sae_lens/training/optim.py", line 38, in get_scheduler
    main_scheduler = _get_main_scheduler(
                     ^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/ml_workspace/SAELens/sae_lens/training/optim.py", line 98, in _get_main_scheduler
    raise ValueError(f"Unsupported scheduler: {scheduler_name}")
ValueError: Unsupported scheduler: constantwithwarmup

System Info
Describe the characteristic of your environment:

Installed from source via poetry install
Ubuntu 22.04
Python 3.11.9

Checklist

[x] I have checked that there is no similar issue in the repo (required)

I have the same issue. It seems lr_scheduler_name should be set to "constant" (warm up is added to any scheduler as long as lr_warm_up_steps != 0)

Apologies. The docs got out of date. I think the training tutorial has up to date hyperparameter examples. @dtch1997 do you think you could take a moment to make a PR to put those hyper pars in the docs? (and maybe add a note in that tutorial about trying to keep them in sync with the docs).


better error handling [nim-eth-p2p]

[this is in relation to https://github.com/status-im/nimbus/issues/198]

changed the logging level of some messages that should not appear by
default in a debug build
exceptions in persistBlocks() are gracefully recovered from
fixed the handling of out of order blocks
dropped work items with fewer blocks than what we requested
getBestBlockNumber(): log and allow exceptions from getBlockHeaders()
obtainBlocksFromPeer(): moved here the check for block numbers being
in sequence (from nimbus/p2p/chain.nim)
gracefully deal with exceptions in returnWorkItem()
sendMsg(): log and allow exceptions, because catching them at the
caller's side seems too complicated. I don't see a mechanism for
retrying to send these messages.

Most exceptions converted to error codes.


Can't resolve 'flowbite-datepicker-1.3.0'

The project fails to compile
./node_modules/flowbite/lib/esm/components/datepicker/index.js:12:0-121 - Error: Module not found: Error: Can't resolve 'flowbite-datepicker-1.3.0' i

I fixed the issue with the import. There may be a better way than installing the library and adding it to the angular.json configuration.


[WFLY-5599] Preliminary changes to expose security domains as Elytron realms.

:exclamation: Commits from this branch are used in numerous other topic branches so please do not cherry-pick or rebase. :exclamation:
This is a minimal PR to make WildFly compatible with the changes in the following core PR https://github.com/wildfly/wildfly-core/pull/1764

Linux with security manager Build 5045 outcome was FAILURE using a merge of 188beb35f787ecbdf575e83a2da2ddd1b74a1c65
Summary: Tests passed: 411, ignored: 38; compilation error: Compiler (new) Build time: 00:02:37

Linux Build 10297 outcome was FAILURE using a merge of 188beb35f787ecbdf575e83a2da2ddd1b74a1c65
Summary: Tests failed: 3 (3 new), passed: 411, ignored: 38; compilation error: Compiler (new) Build time: 00:03:05
Failed tests
org.jboss.as.test.compat.jpa.eclipselink.EclipseLinkSharedModuleProviderTestCase: java.lang.RuntimeException: Arquillian has previously been attempted initialized, but failed. See cause for previous exception
	at org.jboss.as.arquillian.container.DistributionContainerConfiguration.validate(DistributionContainerConfiguration.java:99)
	at org.jboss.as.arquillian.container.managed.ManagedContainerConfiguration.validate(ManagedContainerConfiguration.java:69)

org.jboss.as.test.compat.jpa.hibernate.HibernateJarsInDeploymentTestCase: 	at org.jboss.as.arquillian.container.DistributionContainerConfiguration.validate(DistributionContainerConfiguration.java:99)
	at org.jboss.as.arquillian.container.managed.ManagedContainerConfiguration.validate(ManagedContainerConfiguration.java:69)

org.jboss.as.test.compat.jpa.openjpa.OpenJPASharedModuleProviderTestCase: java.lang.RuntimeException: Arquillian has previously been attempted initialized, but failed. See cause for previous exception
	at org.jboss.as.arquillian.container.DistributionContainerConfiguration.validate(DistributionContainerConfiguration.java:99)
	at org.jboss.as.arquillian.container.managed.ManagedContainerConfiguration.validate(ManagedContainerConfiguration.java:69)

Windows Build 5331 outcome was FAILURE using a merge of 188beb35f787ecbdf575e83a2da2ddd1b74a1c65
Summary: Tests passed: 414, ignored: 38; compilation error: Compiler (new) Build time: 00:05:34

Closing this one as the core update no longer breaks WildFly so we don't need the option of an intermediate commit.


Broken DNS Info Command for Certain Domains

dnsinfo overdrivenetworks.com

Works... at home on debug.

Cannot reproduce


Changes Required

Suggestions
You have done a pretty good job but still, there are things you should work on.

It would be better if you aligned your texts as their respective parent element.
On the mobile version I could see some text overflow I suggest you fix those as it causes the page to be hard to read.

Hi, thank you for these helpful suggestions. 👍
I made the required changes:
Before correction:

[x] aligned the text as their respective parent element (aligned the first summit-description in the list-summits)

After correction:

[x] On the mobile version I could see some text overflow I suggest you fix those as it causes the page to be hard to read. (Add background to the header in the mobile version)
Before correction:

After correction:

Don't hesitate to open an issue if you have any other issues, I will be grateful.


Add PowerShell, CLI, REST API, ARM Template options for configuring Activity Log Diagnostics Setting

Existing guidance is for creating a Diagnostics Setting for a particular resource only not the Azure Activity Log

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 624551f7-b873-face-6b12-f3df50b4fc6d
Version Independent ID: 41954f53-8423-762f-3368-00be2d3fb2b2
Content: Collect Azure Activity log with diagnostic settings (preview) - Azure Monitor
Content Source: articles/azure-monitor/platform/diagnostic-settings-subscription.md
Service: azure-monitor
Sub-service: logs
GitHub Login: @bwren
Microsoft Alias: bwren

@scm9nwr Thanks for the Comment. We are actively investigating and will get back to you soon.

Please provide also an Azure Policy for this scenario to send the logs to Log Analytics. This would be awesome!

Priority #1 would be for Subscriptions :)

Assigning @bwren for visibility, review, insights and for document enhancement considerations.

We are working to enable ARM Templates and REST ASAP, and CLI and Powershell will follow soon after. The review and publishing process can take a few months.

Going to close this one since I think all questions have been answered.  We do have the document updates in our plans.
#please-close


Initial mmsegmentation support

Initial mmsegmentation support. This is a work-in-progress.
Will populate this PR as we go!

[x] Integration can be used end-to-end with DeepLabV3 and SegFormer
[x] W&B integration for semantic segmentation
[x] New checkpoint export/import support (this one might have to wait as feature was added after the fork)
[x] Custom CFG like mmdet implementation (might already be done, but should be tested)
[x] Properly implement "hack" to avoid Pytorch doing parallel training by default in utils.py
[x] Interpretation loop (currently using mmdet's)
[x] Updated semantic segmentation notebook to showcase mmsegmentation models

To figure out:

[x] Splitter / Trainable Params / param_groups
[x] Could we use fastai's Dice and other metrics (if fastai is a hard dependency)? Will keep current implementation

To add in an upcoming PR:

[ ] Lightning integration
[ ] CPU support
[ ] Tests for mmsegmentation models

Heeey nicjac! This is a big and a very very helpful PR, we'll try to assist you as much as we can, feel free to ask for help at any point =)

Hi @nicjac any update here?
cc @ai-fast-track @fstroth @potipot

Hi @nicjac any update here? cc @ai-fast-track @fstroth @potipot

Yep! We've been using this implementation internally for a while, but got sidetracked by other priorities.
Work has resumed, will update the PR and make sure it can be merged.

This looks good to go to me!

I cannot preview the notebooks/getting_started_semantic_segmentation.ipynb notebook.
Is this how it looks like?

Out of curiosity, how do you handle the end2end inference flow with masks (e.g. return the mask on the original image, not the transformed one)?
The current implementation of end2end_detect (here) supports only bboxes.

Out of curiosity, how do you handle the end2end inference flow with masks (e.g. return the mask on the original image, not the transformed one)? The current implementation of end2end_detect (here) supports only bboxes.

Unfortunately, it does not just yet. I have a very hacky implementation for prototyping purpose but a proper implementation will be part of a refinement PR. It shouldn't take too long as we do need this for our work as well!

Out of curiosity, how do you handle the end2end inference flow with masks (e.g. return the mask on the original image, not the transformed one)? The current implementation of end2end_detect (here) supports only bboxes.

Unfortunately, it does not just yet. I have a very hacky implementation for prototyping purpose but a proper implementation will be part of a refinement PR. It shouldn't take too long as we do need this for our work as well!

I am very interested in this one, so I can definitely take care of it too.


Errors Updating Options

Describe the bug
When I first started, none of the panels were loading data. I then edited each panel to select Datasource as Postgressql (instead of the default Teslamate) and everything is loading perfectly.  But I keep getting the following errors (or similar) as I load Grafana panels.  Where do I set the base_url and where do I edit to include the appropriate data source.   (I had originally used 'teslamate' as the database name in the postgres configuration).  Thanks for your help.

Closed as no activity


Lucas theorem in java

💥 Proposal
(A clear and concise description of what the proposal is.)
Lucas theorem in java
Have you read the Contributing Guidelines on Pull Requests?
(Write your answer here.)
yes

/assign


Name of output file should be allowed to be overridden

Provide a means to completely override the name of the output file.
Will need to consider the case when multiple output files are going to be generated. Maybe they can be numbered?

Done


Typespec for types field of Changeset.t() is incomplete

Environment

Elixir version (elixir -v): Elixir 1.10.0 (compiled with Erlang/OTP 22)
Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.): PostgreSQL 11.6
Ecto version (mix deps): 3.5.5
Database adapter and version (mix deps): postgrex 0.15.7
Operating system: MacOS 10 Mojave

Current behavior
Using hammox to add typechecking for my Mox mocks, I'm getting a type mismatch error in Ecto.Changeset.t() on the types field.  I assume Dialyzer would report the same issue, but we're not currently using it on this project, so I don't have an easy way to check that.
The most relevant message from the output (slightly anonymized) is:
 Map value {:assoc, %Ecto.Association.BelongsTo{cardinality: :one, defaults: [], field: :account, on_cast: nil, on_replace: :raise, ordered: false, owner: User, owner_key: :account_id, queryable: Account, related: Account, related_key: :id, relationship: :parent, unique: true, where: []}} for key :account does not match map value type Ecto.Type.t().

The typespec for Ecto.Changeset.t() says that the types field has type nil | %{atom => Ecto.Type.t}, but the {:assoc, <association>} tuple (and from looking at the code, I think also the {:embed, <???>} tuple don't match Ecto.Type.t().
Expected behavior
A changeset should correctly match the Ecto.Changeset.t() typespec.
Analysis
I'm not terribly familiar with this codebase, but I don't think the :assoc/:embed tuples should be part of Ecto.Type.t()'s definition.  Instead, I think the typespec for the types field should be something like:
nil | %{atom => Ecto.Type.t | {:assoc, Ecto.Association.t()} | {:embed, ???}

I'm not sure what the correct type for the :embed tuple should be.
If this sounds like the right direction to go, I'd be happy to work on a PR, but would need some guidance on the correct typespec for the :embed tuple.
Thanks so much for a great library!

Hi there, @randycoulman! I think I fixed it on #3505, but I didn't manage to reproduce your issue. Can you check if it works and give an okay in the PR once you do it? :)


Test an Announce

Same as fediverse-devnet/feditest-tests-fediverse#17, except Announce (ie repost) instead of Like.

Split off #150 .


allow INSERT statement with a query

Add support for INSERT INTO table  ( col..) SELECT ...
Care must be taken to ensure that datatypes from the query match up with the insert table.
Current workaround for this is COPY TO/FROM, but avoiding intermediate the transfer and file write would be faster and more convenient.

Related with this issue, i'm curious how mapd_server handles a SELECT that returns data of a size larger than system memory. On my VM with 14GB free memory, query "SELECT * from trips" failed due to OOM. Also failed is another query "copy (select * from trips) to '/mnt/tmp/trips.csv'", with similar INFO logs as below shows. The table trips has 512M rows, using approximately 47GB disk space.
Is there any query option that can work around this limitation? If no, does it mean that, regardless intermediate file write, a COPY TO query of data larger than available memory will fail?
If that is the case, what is the complexity of a change to current design to make the part of code that returns query result to work in a streaming manner?
I am curious about a parameter use_resume_param in function NativeCodegen.cpp:bind_pos_placeholders and what it tries to "resume". Somehow with gdb i can't trace into  RuntimeFunctions.bc to learn the effect of the parameter and how related LLVM code works. Any advice in this regard will be appreciated.
E1017 20:09:34.659145  8212 MapDHandler.cpp:139] This build isn't CUDA enabled, will run on CPU
I1017 20:09:34.671910  8212 DataMgr.cpp:120] cpuSlabSize is 4096M
I1017 20:09:34.698573  8212 FileMgr.cpp:173] Completed Reading table's file metadata, Elasped time : 11ms Epoch: 0 files read: 0 table location: '/mnt/trip/mapd/data/mapd_data/table_0_0/'
I1017 20:09:34.707365  8212 Calcite.cpp:156] Creating Calcite Handler,  Calcite Port is 9093 base data dir is /mnt/trip/mapd/data
I1017 20:09:34.707384  8212 Calcite.cpp:95] Running calcite server as a daemon
I1017 20:09:35.374925  8212 Calcite.cpp:124] Calcite server start took 600 ms
I1017 20:09:35.374971  8212 Calcite.cpp:125] ping took 18 ms
I1017 20:09:35.380465  8212 Calcite.cpp:281] [{"name":"Tan","ret":"double","args":["double"]},{"name":"Truncate__","ret":"float","args":["float","i32"]},{"name":"ln","ret":"double","args":["double"]},{"name":"distance_in_meters__","ret":"double","args":["float","float","float","float"]},{"name":"Floor__","ret":"float","args":["float"]},{"name":"Floor__2","ret":"i32","args":["i32"]},{"name":"Floor__3","ret":"i64","args":["i64"]},{"name":"Truncate","ret":"double","args":["double","i32"]},{"name":"Floor__1","ret":"i16","args":["i16"]},{"name":"radians","ret":"double","args":["double"]},{"name":"degrees","ret":"double","args":["double"]},{"name":"Ceil__","ret":"float","args":["float"]},{"name":"ln__","ret":"double","args":["float"]},{"name":"approx_distance_in_meters","ret":"double","args":["float","float","float","float"]},{"name":"Ceil__2","ret":"i32","args":["i32"]},{"name":"Ceil__1","ret":"i16","args":["i16"]},{"name":"Ceil__3","ret":"i64","args":["i64"]},{"name":"power","ret":"double","args":["double","double"]},{"name":"conv_4326_900913_x","ret":"double","args":["double"]},{"name":"conv_4326_900913_y","ret":"double","args":["double"]},{"name":"Atan","ret":"double","args":["double"]},{"name":"Floor","ret":"double","args":["double"]},{"name":"Log10","ret":"double","args":["double"]},{"name":"Truncate__1","ret":"i16","args":["i16","i32"]},{"name":"Truncate__2","ret":"i32","args":["i32","i32"]},{"name":"Log","ret":"double","args":["double"]},{"name":"Truncate__3","ret":"i64","args":["i64","i32"]},{"name":"Cos","ret":"double","args":["double"]},{"name":"Log__","ret":"double","args":["float"]},{"name":"Log10__","ret":"double","args":["float"]},{"name":"Asin","ret":"double","args":["double"]},{"name":"Cot","ret":"double","args":["double"]},{"name":"Tan__","ret":"double","args":["float"]},{"name":"distance_in_meters","ret":"double","args":["double","double","double","double"]},{"name":"round_to_digit","ret":"double","args":["double","i32"]},{"name":"Atan2","ret":"double","args":["double","double"]},{"name":"Acos","ret":"double","args":["double"]},{"name":"pi","ret":"double","args":[]},{"name":"Sin","ret":"double","args":["double"]},{"name":"Ceil","ret":"double","args":["double"]},{"name":"Exp","ret":"double","args":["double"]}]
E1017 20:09:35.403663  8212 MapDHandler.cpp:171] No GPUs detected, falling back to CPU mode
I1017 20:09:35.403697  8212 MapDHandler.cpp:180] Started in CPU mode
I1017 20:10:05.424568  8275 MapDHandler.cpp:311] User mapd connected to database mapd
I1017 20:10:05.472877  8275 MapDHandler.cpp:2206] copy (select * from trips) to '/mnt/tmp/trips.csv';
I1017 20:10:05.477603  8275 MapDHandler.cpp:2234] passing query to legacy processor
I1017 20:10:05.489598  8275 Calcite.cpp:247] User mapd catalog mapd sql 'select * from trips;'
I1017 20:13:25.988349  8278 FileMgr.cpp:173] Completed Reading table's file metadata, Elasped time : 199575ms Epoch: 125234 files read: 506 table location: '/mnt/trip/mapd/data/mapd_data/table_1_1/'
I1017 20:13:38.456360  8278 Catalog.cpp:1511] Instantiating Fragmenter for table trips took 212085ms
I1017 20:13:38.626971  8278 Catalog.cpp:1557] Time to load Dictionary 1 was 170ms
I1017 20:13:38.763283  8278 Catalog.cpp:1557] Time to load Dictionary 2 was 136ms
I1017 20:13:38.881168  8278 Catalog.cpp:1557] Time to load Dictionary 3 was 117ms
I1017 20:13:38.990425  8278 Catalog.cpp:1557] Time to load Dictionary 4 was 109ms
I1017 20:13:39.280971  8275 Calcite.cpp:260] Time in Thrift 13 (ms), Time in Java Calcite server 213778 (ms)
I1017 20:13:39.716213 10892 BufferMgr.cpp:266] ALLOCATION slab of 8388608 pages<PHONE_NUMBER>B) created in 0 ms CPU_MGR:0
I1017 20:14:30.122083 10893 BufferMgr.cpp:266] ALLOCATION slab of 8388608 pages<PHONE_NUMBER>B) created in 0 ms CPU_MGR:0
I1017 20:15:30.326671 10895 BufferMgr.cpp:266] ALLOCATION slab of 6633401 pages<PHONE_NUMBER>B) created in 0 ms CPU_MGR:0
E1017 20:16:21.526213 10898 BufferMgr.cpp:370] ALLOCATION failed to find 300000256B throwing out of memory CPU_MGR:0
E1017 20:16:21.682365 10899 BufferMgr.cpp:370] ALLOCATION failed to find 503252992B throwing out of memory CPU_MGR:0
E1017 20:16:21.683790 10894 BufferMgr.cpp:370] ALLOCATION failed to find 300000256B throwing out of memory CPU_MGR:0
E1017 20:16:21.685151 10893 BufferMgr.cpp:370] ALLOCATION failed to find 600000000B throwing out of memory CPU_MGR:0
E1017 20:16:21.685667 10892 BufferMgr.cpp:370] ALLOCATION failed to find 600000000B throwing out of memory CPU_MGR:0
E1017 20:16:21.686103 10895 BufferMgr.cpp:370] ALLOCATION failed to find 300000256B throwing out of memory CPU_MGR:0
I1017 20:16:24.503365 10897 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 0 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:16:29.814640 10897 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 585938 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:16:40.076355 10897 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 3222659 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:16:52.534536 10897 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 1757813 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:16:57.187934 10897 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4687503 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:17:06.715833 10897 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 5859378 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:17:17.180187 10897 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 0 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:17:27.879951 10897 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 1171875 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:17:38.307312 10897 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 2343750 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:17:49.696372 11245 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 2343751 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:17:54.284348 11245 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 7031253 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:17:58.744581 11247 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 7617191 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:18:04.263002 11248 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 3515625 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:18:09.654181 11249 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 4101563 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:18:14.792775 11250 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 4687501 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:18:25.347380 11246 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 5859377 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:18:34.596098 11247 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 7031253 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:18:41.665531 11247 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 585938 Number pages requested 585938 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:18:50.835289 11250 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 2835210 Number pages requested 585938 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:18:55.884757 11246 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 3421148 Number pages requested 585938 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:19:07.918150 11248 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 4687503 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:19:12.988420 11247 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 5273441 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:19:27.930819 11250 BufferMgr.cpp:375] ALLOCATION failed to find 150000128B free. Forcing Eviction. Eviction start 1757814 Number pages requested 292969 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:19:30.514434 11250 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 0 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:19:35.177902 11245 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 585938 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:19:39.440032 11245 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 1171876 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
E1017 20:19:48.355754 11248 BufferMgr.cpp:370] ALLOCATION failed to find 600000000B throwing out of memory CPU_MGR:0
E1017 20:19:48.418704 11247 BufferMgr.cpp:370] ALLOCATION failed to find 600000000B throwing out of memory CPU_MGR:0
I1017 20:19:54.316274 11251 BufferMgr.cpp:375] ALLOCATION failed to find 251626496B free. Forcing Eviction. Eviction start 7617191 Number pages requested 491458 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:19:58.017437 11251 BufferMgr.cpp:375] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 3080938 Number pages requested 982916 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:20:06.792656 11251 BufferMgr.cpp:375] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 0 Number pages requested 982916 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:20:15.761013 11246 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 2050783 Number pages requested 585938 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:20:20.932322 11246 BufferMgr.cpp:375] ALLOCATION failed to find 150000128B free. Forcing Eviction. Eviction start 7031253 Number pages requested 292969 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:20:22.847684 11246 BufferMgr.cpp:375] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 4300055 Number pages requested 585938 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:20:27.118314 11246 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4687503 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
E1017 20:20:36.944784 11246 BufferMgr.cpp:370] ALLOCATION failed to find 600000000B throwing out of memory CPU_MGR:0
I1017 20:20:36.953239 11250 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4687503 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
E1017 20:20:47.808964 11250 BufferMgr.cpp:370] ALLOCATION failed to find 600000000B throwing out of memory CPU_MGR:0
I1017 20:20:47.839965 11245 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4687501 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:20:59.088346 11249 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 2636721 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:21:09.551349 11251 BufferMgr.cpp:375] ALLOCATION failed to find 251626496B free. Forcing Eviction. Eviction start 5859376 Number pages requested 491458 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:21:14.014973 11251 BufferMgr.cpp:375] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 4300055 Number pages requested 982916 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:21:22.807905 11249 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4687503 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
E1017 20:21:33.186449 11249 BufferMgr.cpp:370] ALLOCATION failed to find 600000000B throwing out of memory CPU_MGR:0
I1017 20:21:33.188721 11251 BufferMgr.cpp:375] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 5282971 Number pages requested 982916 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:21:42.195564 11251 BufferMgr.cpp:375] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 0 Number pages requested 982916 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:21:50.880193 11251 BufferMgr.cpp:375] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 3222658 Number pages requested 982916 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:22:04.150955 11245 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 982916 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:22:13.288806 11251 BufferMgr.cpp:375] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 2154791 Number pages requested 982916 Best Eviction Start Slab 2 CPU_MGR:0
I1017 20:22:21.842098 11245 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4205574 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1017 20:22:31.569769 11245 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 2343751 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:22:40.674849 11245 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 3515626 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1017 20:22:49.867633 11245 BufferMgr.cpp:375] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 7031253 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
E1017 20:22:59.365795  8275 MapDHandler.cpp:2306] Exception: Other error: code -16384
I1017 20:22:59.465265  8275 MapDHandler.cpp:321] User mapd disconnected from database mapd

Yes, that's the deal.  Currently the working set for the query needs to fit in GPU (or CPU) memory.  There's room for several straightforward optimizations for large projection queries that would improve the efficiency to allow much larger results.  Maybe this is a project for you one of these days, though I'd have in mind several other things ahead of this.

@billmaimone Would you mind giving a hint on one or two of the several straightforward optimizations in your mind? I feel it not simply an optimization, because current code acts more like a scatter-gather and lump-sum model while to enable it to stream larger datasets some parts of the code need to collaborate like synchronized pipelines or like coroutines.

Support for this type of workload can be added without fundamental changes to the way we execute queries. We already have the ability to consume the input tables fragment-by-fragment; to minimize memory usage, we'd have to unpin the chunks in a fragment once it's been handled. Also, the output for each fragment can be written to disk right away instead of keeping it in memory. Ultimately it's a matter of priorities; I'm not sure of the relative importance of being the go-to system for what doesn't sound like an analytic workload vs other things we can improve.
Also, co-routines might be a convenient way to think about the problem, but ultimately they have to be lowered to native code and runtime support. I'm not sure we need that level of sophistication just to get reasonable overlap between data fetching and computation.

@asuhan. Agreed. Originally i thought the largest memory consumer is intermediate result, so i thought co-routine may help. Afterward i found the largest memory consumer is the file chunk buffers. So i've been probing usage of chunk buffers...

You can't breakpoint into RuntimeFunctions.bc because that code is not directly linked into the server executable. Instead, the LLVM IR from it is inserted inline in the generated code. Also, the functions you've mentioned are actually replaced by real versions which are different between CPU and GPU when the query is compiled to native code.

@fleapapa The intermediate memory for very big projection queries (of the select all rows and columns from the table kind) can be an issue as well. The good news is it can be optimized dramatically, to the point that a full co-routine model would be overkill on all but the most severely memory constrained systems. According to my back-of-the-envelope math, we could probably work with ~1 GB of memory without fundamental changes to the way the query executes, just by carefully implementing the ideas I've mentioned.

Quoted:
We already have the ability to consume the input tables fragment-by-fragment; to minimize memory usage, we'd have to unpin the chunks in a fragment once it's been handled.
As traced, the problem with this scheme is that for each fragment there're usually several threads (eg. 7 threads on my VM), each of which is responsible for loading a few columns. Since they run in parallel, an always-happening scenario with my test case is, as shown below, that there is always at least one thread that can't find a buffer big enough and can't wait for buffers unpinned by other threads before it throws OOM exception, which causes the query to fail.
Slab St.Page   Pages  Touch
   0       0  585938 GAP:       0      0 PC:  1 USED - Chunk: 1,1,1,0,
   0  585938  585938 GAP:       0      1 PC:  1 USED - Chunk: 1,1,1,1,
   0 1171876  585938 GAP:       0      2 PC:  1 USED - Chunk: 1,1,2,1,
   0 1757814  585938 GAP:       0      3 PC:  1 USED - Chunk: 1,1,1,3,
   0 2343752  585938 GAP:       0      4 PC:  1 USED - Chunk: 1,1,2,3,
   0 2929690  585938 GAP:       0      5 PC:  1 USED - Chunk: 1,1,1,5,
   0 3515628  585938 GAP:       0      6 PC:  1 USED - Chunk: 1,1,2,5,
   0 4101566  585938 GAP:       0      7 PC:  1 USED - Chunk: 1,1,3,5,
   0 4687504  585938 GAP:       0      8 PC:  1 USED - Chunk: 1,1,1,2,
   0 5273442  585938 GAP:       0      9 PC:  1 USED - Chunk: 1,1,2,2,
   0 5859380  585938 GAP:       0     10 PC:  1 USED - Chunk: 1,1,3,2,
   0 6445318  585938 GAP:       0     11 PC:  1 USED - Chunk: 1,1,3,3,
   0 7031256  292969 GAP:       0     12 PC:  1 USED - Chunk: 1,1,4,3,
   0 7324225  585938 GAP:       0     13 PC:  1 USED - Chunk: 1,1,2,0,
   0 7910163  292969 GAP:       0     16 PC:  1 USED - Chunk: 1,1,4,1,
   0 8203132  185476 GAP:       0      0 FREE
Slab St.Page   Pages  Touch
   1       0  585938 GAP:       0     14 PC:  1 USED - Chunk: 1,1,3,0,
   1  585938  585938 GAP:       0     15 PC:  1 USED - Chunk: 1,1,3,1,
   1 1171876  585938 GAP:       0     17 PC:  1 USED - Chunk: 1,1,5,1,
   1 1757814  491458 GAP:       0     18 PC:  1 USED - Chunk: 1,1,1,6,
   1 2249272  491458 GAP:       0     19 PC:  1 USED - Chunk: 1,1,2,6,
   1 2740730  491458 GAP:       0     20 PC:  1 USED - Chunk: 1,1,3,6,
   1 3232188  292969 GAP:       0     21 PC:  1 USED - Chunk: 1,1,4,0,
   1 3525157  585938 GAP:       0     22 PC:  1 USED - Chunk: 1,1,5,0,
   1 4111095 1171875 GAP:       0     23 PC:  1 USED - Chunk: 1,1,6,0,
   1 5282970 1171875 GAP:       0     24 PC:  1 USED - Chunk: 1,1,6,1,
   1 6454845  585938 GAP:       0     25 PC:  1 USED - Chunk: 1,1,5,3,
   1 7040783 1171875 GAP:       0     26 PC:  1 USED - Chunk: 1,1,6,3,
   1 8212658  175950 GAP:       0      0 FREE
Slab St.Page   Pages  Touch
   2       0  245729 GAP:       0     27 PC:  1 USED - Chunk: 1,1,4,6,
   2  245729  491458 GAP:       0     28 PC:  1 USED - Chunk: 1,1,5,6,
   2  737187  292969 GAP:       0     29 PC:  1 USED - Chunk: 1,1,4,2,
   2 1030156 1171875 GAP:       0     30 PC:  1 USED - Chunk: 1,1,7,0,
   2 2202031  292969 GAP:       0     31 PC:  1 USED - Chunk: 1,1,8,0,
   2 2495000  585938 GAP:       0     32 PC:  1 USED - Chunk: 1,1,9,0,
   2 3080938 1171875 GAP:       0     33 PC:  1 USED - Chunk: 1,1,10,0,
   2 4252813  585938 GAP:       0     34 PC:  1 USED - Chunk: 1,1,1,4,
   2 4838751  982916 GAP:       0     35 PC:  1 USED - Chunk: 1,1,6,6,
   2 5821667  585938 GAP:       0     36 PC:  1 USED - Chunk: 1,1,5,2,
   2 6407605  225796 GAP:       0      0 FREE

Yes, we'd have to throttle the execution (a memory quota per thread), which ultimately means that our optimization on the query execution side is less useful for this type of workloads. This is the reason we didn't really focus on them -- we have less of a comparative advantage for low-memory scenarios vs the ones we've already focused on.

I guess for this kind of work load users usually look for entirety of query result than query performance.
For example, when i backup/copy my BIG tables, i am more concerned about whether all my data is written to destination storage than about how long it will take.

Yes, backup is a potential application for it. At the same time, backup can (and probably should) be implemented as a separate feature, it doesn't need to be in the roundabout way of running a SELECT * query. At the same time, there's definitely room to better accommodate certain non-analytic queries for well-defined purposes.

I might have missing another application -- ETL :)
ETL jobs usually query/aggregate multiple BIG tables to generate a BIG datasets for offline analytics and model training.

@asuhan,
Yes, we'd have to throttle the execution (a memory quota per thread),
Per your hint, i took an easier path. Instead of throttling memory usage per thread, i sequentially ran the threads. This time, all fragments were fetched, but then execution halted at GroupByAndAggregate.cpp:138. To proceed it needs a big "group by" buffer of 60 GB (the table uses 47 GB disk space). I could go get 4 x 16GB DIMM but what if my table is 1TB or larger? I don't think my server can install TBs or even 1TB memory...

That's where the dramatical reduction of memory usage for projection output comes into play. For big projection queries, a bitmap layout would be orders of magnitude smaller, you wouldn't need anything close to 60GB. More like 600MB.

Is the "dramatical reduction" in the code? A quick pointer or hint will be appreciated.

No, it's not yet, but we know how to build it and I think we also have an internal task for it. It's not particularly difficult, but it touches all the executor parts (some code generation, the result set etc).

@asuhan. I have a long question of Executor::executePlanWithoutGroupBy().
My query is "COPY (SELECT * FROM trips) TO 'trips.csv'". The table trips has ~0.5B rows. From gdb, after the first fragment (75000000 rows x 14 columns) was read into memory, the first call of executePlanWithoutGroupBy() came out with a result set which has only one value:
(gdb) p reduced_outs
$28 = std::vector of length 1, capacity 1 = {75000000}

I thought the result set would be a 75M x 14 block of values. My query has no aggregation, so i'm confused why the function ends up with only a single value. Please shed some light here.

I doubt that method should be even hit by the query. Are you certain that the source is not newer / older than the executable? Also, what's in the output trips.csv file, does the operation work?

No, the operation doesn't work yet. The file trips.csv isn't even created, because after 7th call of the method (because the table has 7 fragments), the code always ends up with a failure to allocate 60GB memory.
The method was indeed hit, as shown below.
(gdb) bt
#0  Executor::executePlanWithoutGroupBy (this=0x7fffd401edb0, ra_exe_unit=..., compilation_result=..., hoist_literals=true, results=..., target_exprs=std::vector of length 1, capacity 1 = {...},
    device_type=ExecutorDeviceType::CPU, col_buffers=std::vector of length 1, capacity 1 = {...}, query_exe_context=0x7fffd40dc620, num_rows=std::vector of length 1, capacity 1 = {...},
    frag_offsets=std::vector of length 1, capacity 1 = {...}, frag_stride=1, data_mgr=0x388f6e0, device_id=0, start_rowid=0, num_tables=1, render_allocator_map=0x0) at /data/github/mapd-core/QueryEngine/Execute.cpp:1845
#1  0x00000000012cc3d3 in Executor::ExecutionDispatch::runImpl (this=0x7fffdf101f90, chosen_device_type=ExecutorDeviceType::CPU, chosen_device_id=0, options=..., frag_ids=std::vector of length 1, capacity 1 = {...}, ctx_idx=0,
    rowid_lookup_key=-1) at /data/github/mapd-core/QueryEngine/ExecutionDispatch.cpp:333
#2  0x00000000012cdcf0 in Executor::ExecutionDispatch::run (this=0x7fffdf101f90, chosen_device_type=ExecutorDeviceType::CPU, chosen_device_id=0, options=..., frag_ids=std::vector of length 1, capacity 1 = {...}, ctx_idx=0,
    rowid_lookup_key=-1) at /data/github/mapd-core/QueryEngine/ExecutionDispatch.cpp:513
#3  0x000000000123b8ba in Executor::<lambda(ExecutorDeviceType, int, const std::vector<std::pair<int, std::vector<long unsigned int, std::allocator<long unsigned int> > >, std::allocator<std::pair<int, std::vector<long unsigned int, std::allocator<long unsigned int> > > > >&, size_t, int64_t)>::operator()(ExecutorDeviceType, int, const std::vector<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > >, std::allocator<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > > > > &, size_t, int64_t) const (__closure=0x7fffd4221220, chosen_device_type=ExecutorDeviceType::CPU, chosen_device_id=0,
    frag_ids=std::vector of length 1, capacity 1 = {...}, ctx_idx=0, rowid_lookup_key=-1) at /data/github/mapd-core/QueryEngine/Execute.cpp:868
#4  0x0000000001273b9f in std::_Function_handler<void(ExecutorDeviceType, int, const std::vector<std::pair<int, std::vector<long unsigned int, std::allocator<long unsigned int> > >, std::allocator<std::pair<int, std::vector<long unsigned int, std::allocator<long unsigned int> > > > >&, long unsigned int, long int), Executor::executeWorkUnit(int32_t*, size_t&, bool, const std::vector<InputTableInfo>&, const RelAlgExecutionUnit&, const CompilationOptions&, const ExecutionOptions&, const Catalog_Namespace::Catalog&, std::shared_ptr<RowSetMemoryOwner>, RenderAllocatorMap*, bool)::<lambda(ExecutorDeviceType, int, const std::vector<std::pair<int, std::vector<long unsigned int, std::allocator<long unsigned int> > >, std::allocator<std::pair<int, std::vector<long unsigned int, std::allocator<long unsigned int> > > > >&, size_t, int64_t)> >::_M_invoke(const std::_Any_data &, <unknown type in /mnt/mapd-core/bin/mapd_server, CU 0xa37f35, DIE 0xbc6b51>, <unknown type in /mnt/mapd-core/bin/mapd_server, CU 0xa37f35, DIE 0xb7f2bf>, const std::vector<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > >, std::allocator<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > > > > &, <unknown type in /mnt/mapd-core/bin/mapd_server, CU 0xa37f35, DIE 0xb56ed8>, <unknown type in /mnt/mapd-core/bin/mapd_server, CU 0xa37f35, DIE 0xb5a203>) (__functor=..., __args#0=<unknown type in /mnt/mapd-core/bin/mapd_server, CU 0xa37f35, DIE 0xbc6b51>, __args#1=<unknown type in /mnt/mapd-core/bin/mapd_server, CU 0xa37f35, DIE 0xb7f2bf>,
    __args#2=std::vector of length 1, capacity 1 = {...}, __args#3=<unknown type in /mnt/mapd-core/bin/mapd_server, CU 0xa37f35, DIE 0xb56ed8>,
    __args#4=<unknown type in /mnt/mapd-core/bin/mapd_server, CU 0xa37f35, DIE 0xb5a203>) at /usr/include/c++/6/functional:1731
#5  0x0000000001283716 in std::function<void (ExecutorDeviceType, int, std::vector<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > >, std::allocator<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > > > > const&, unsigned long, long)>::operator()(ExecutorDeviceType, int, std::vector<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > >, std::allocator<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > > > > const&, unsigned long, long) const (this=0x7fffdf101e20, __args#0=ExecutorDeviceType::CPU, __args#1=0, __args#2=std::vector of length 1, capacity 1 = {...}, __args#3=0,
    __args#4=-1) at /usr/include/c++/6/functional:2127
#6  0x0000000001240552 in Executor::dispatchFragments(std::function<void (ExecutorDeviceType, int, std::vector<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > >, std::allocator<std::pair<int, std::vector<unsigned long, std::allocator<unsigned long> > > > > const&, unsigned long, long)>, Executor::ExecutionDispatch const&, ExecutionOptions const&, bool, std::map<int, std::deque<Fragmenter_Namespace::FragmentInfo, std::allocator<Fragmenter_Namespace::FragmentInfo> > const*, std::less<int>, std::allocator<std::pair<int const, std::deque<Fragmenter_Namespace::FragmentInfo, std::allocator<Fragmenter_Namespace::FragmentInfo> > const*> > >&, unsigned long, std::condition_variable&, std::mutex&, std::unordered_set<int, std::hash<int>, std::equal_to<int>, std::allocator<int> >&, int&) (this=0x7fffd401edb0, dispatch=..., execution_dispatch=..., eo=..., is_agg=true,
    selected_tables_fragments=std::map with 1 element = {...}, context_count=14, scheduler_cv=..., scheduler_mutex=..., available_gpus=std::unordered_set with 0 elements, available_cpus=@0x7fffdf101ba4: 14)
    at /data/github/mapd-core/QueryEngine/Execute.cpp:1296
#7  0x000000000123c60e in Executor::executeWorkUnit (this=0x7fffd401edb0, error_code=0x7fffdf1024dc, max_groups_buffer_entry_guess=@0x7fffdf1024e0: 1, is_agg=true, query_infos=std::vector of length 1, capacity 1 = {...},
    ra_exe_unit=..., co=..., options=..., cat=..., row_set_mem_owner=std::shared_ptr (count 5, weak 0) 0x7fffd4033fe0, render_allocator_map=0x0, has_cardinality_estimation=false)
    at /data/github/mapd-core/QueryEngine/Execute.cpp:907
#8  0x0000000001380399 in RelAlgExecutor::getFilteredCountAll (this=0x7fffdf103720, work_unit=..., is_agg=true, co=..., eo=...) at /data/github/mapd-core/QueryEngine/RelAlgExecutor.cpp:1417
#9  0x000000000137f57b in RelAlgExecutor::executeWorkUnit (this=0x7fffdf103720, work_unit=..., targets_meta=std::vector of length 14, capacity 14 = {...}, is_agg=false, co=..., eo=..., render_info=0x0, queue_time_ms=0)
    at /data/github/mapd-core/QueryEngine/RelAlgExecutor.cpp:1309
#10 0x000000000137d522 in RelAlgExecutor::executeProject (this=0x7fffdf103720, project=0x7fffd4032220, co=..., eo=..., render_info=0x0, queue_time_ms=0) at /data/github/mapd-core/QueryEngine/RelAlgExecutor.cpp:1036
#11 0x0000000001377d96 in RelAlgExecutor::executeRelAlgStep (this=0x7fffdf103720, i=0, exec_descs=std::vector of length 1, capacity 1 = {...}, co=..., eo=..., render_info=0x0, queue_time_ms=0)
    at /data/github/mapd-core/QueryEngine/RelAlgExecutor.cpp:379
#12 0x0000000001376346 in RelAlgExecutor::executeRelAlgSeq (this=0x7fffdf103720, exec_descs=std::vector of length 1, capacity 1 = {...}, co=..., eo=..., render_info=0x0, queue_time_ms=0)
    at /data/github/mapd-core/QueryEngine/RelAlgExecutor.cpp:231
#13 0x0000000001374bb3 in RelAlgExecutor::executeRelAlgQueryNoRetry (this=0x7fffdf103720,
    query_ra="{\n  \"rels\": [\n    {\n      \"id\": \"0\",\n      \"relOp\": \"EnumerableTableScan\",\n      \"fieldNames\": [\n        \"medallion\",\n        \"hack_license\",\n        \"vendor_id\",\n        \"rate_code_id\",\n        \"stor"..., co=..., eo=..., render_info=0x0) at /data/github/mapd-core/QueryEngine/RelAlgExecutor.cpp:86
#14 0x000000000137451e in RelAlgExecutor::executeRelAlgQuery (this=0x7fffdf103720,
    query_ra="{\n  \"rels\": [\n    {\n      \"id\": \"0\",\n      \"relOp\": \"EnumerableTableScan\",\n      \"fieldNames\": [\n        \"medallion\",\n        \"hack_license\",\n        \"vendor_id\",\n        \"rate_code_id\",\n        \"stor"..., co=..., eo=..., render_info=0x0) at /data/github/mapd-core/QueryEngine/RelAlgExecutor.cpp:40
#15 0x0000000001183564 in Parser::getResultRows (session=..., select_stmt="select * from trips;", targets=std::vector of length 0, capacity 0) at /data/github/mapd-core/Parser/ParserNode.cpp:1927
#16 0x000000000118c703 in Parser::ExportQueryStmt::execute (this=0x7fffd4016340, session=...) at /data/github/mapd-core/Parser/ParserNode.cpp:2760
---Type <return> to continue, or q <return> to quit---q
Quit

The executable should be in sync with the source code, because each time i rebuilt the executable gdb notified me of symbol reload.

Ok, now I understand what's going on. As you can see in the stack trace, that first ResultSet comes from a RelAlgExecutor::getFilteredCountAll call (and it looks correct). For projection queries on big tables, we first run a COUNT(*) query (with all the filters applied, if any) to find out how big the result should be. So there's nothing concerning / incorrect here.

Why doesn't it simply run SELECT * directly?  Otherwise (especially with tight memory) when it does it needs to reload the fragment from disk.

Or, is it complicated (for me) to bypass getFilteredCountAll?

No, we don't need to load data from disk except for the columns which are used for the filter (none in this case), for the rest we only need the meta data.
Having to start with a uneducated guess and keep reallocating is very expensive compared to running a cheap COUNT(*) query.
No, it's not complicated at all, but for the reasons above you don't want to.

I don't understand 1. My VM has only 14GB free memory. The table needs at least 48GB space, so each fragment needs at least 6GB. That means after 3rd call of the method, the first fragment would have been evicted.

On what grounds do you think that a SELECT COUNT(*) FROM trips needs to load anything but some metadata in the memory? Is there evidence for it?

From the pages loaded and timestamps, you can see the server worked hard to read 48GB data into memory (by many evictions). That's why i think it needs to reload the fragments if the code does not directly perform my query on the first chance.
I1020 21:25:16.984474 23391 MapDHandler.cpp:2206] copy (select * from trips) to '/mnt/tmp/trips.csv';
I1020 21:25:16.987905 23391 MapDHandler.cpp:2234] passing query to legacy processor
I1020 21:25:16.996446 23391 Calcite.cpp:247] User mapd catalog mapd sql 'select * from trips;'
I1020 21:28:44.089020 23394 FileMgr.cpp:173] Completed Reading table's file metadata, Elasped time : 206148ms Epoch: 125234 files read: 506 table location: '/mnt/trip/mapd/data/mapd_data/table_1_1/'
I1020 21:28:56.028679 23394 Catalog.cpp:1511] Instantiating Fragmenter for table trips took 218122ms
I1020 21:28:56.243412 23394 Catalog.cpp:1557] Time to load Dictionary 1 was 214ms
I1020 21:28:56.386852 23394 Catalog.cpp:1557] Time to load Dictionary 2 was 143ms
I1020 21:28:56.520334 23394 Catalog.cpp:1557] Time to load Dictionary 3 was 133ms
I1020 21:28:56.630808 23394 Catalog.cpp:1557] Time to load Dictionary 4 was 110ms
I1020 21:28:56.929828 23391 Calcite.cpp:260] Time in Thrift 28 (ms), Time in Java Calcite server 219905 (ms)
I1020 21:28:57.484974 23391 BufferMgr.cpp:270] ALLOCATION slab of 8388608 pages<PHONE_NUMBER>B) created in 0 ms CPU_MGR:0
I1020 21:29:41.467334 23391 BufferMgr.cpp:270] ALLOCATION slab of 8388608 pages<PHONE_NUMBER>B) created in 12 ms CPU_MGR:0
I1020 21:30:02.858394 23391 ExecutionDispatch.cpp:375] 1 results so far
I1020 21:30:02.858515 23391 ExecutionDispatch.cpp:380] 	 frag 0: 1 rows, 1 cols
I1020 21:30:28.046356 23391 BufferMgr.cpp:270] ALLOCATION slab of 6633420 pages<PHONE_NUMBER>B) created in 0 ms CPU_MGR:0
I1020 21:31:20.059157 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 0 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:31:29.099066 23391 ExecutionDispatch.cpp:375] 2 results so far
I1020 21:31:29.099279 23391 ExecutionDispatch.cpp:380] 	 frag 0: 1 rows, 1 cols
I1020 21:31:29.099329 23391 ExecutionDispatch.cpp:380] 	 frag 1: 1 rows, 1 cols
I1020 21:31:34.358129 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 1171875 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:31:39.039259 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 1757813 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:31:43.808163 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 2636720 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:31:47.103474 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 3222658 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:31:56.545462 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4394533 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:32:07.710490 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 5859377 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:32:11.881705 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 6445315 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:32:21.657599 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 0 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:32:31.904186 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 1171875 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:32:41.506430 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 2343750 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:32:51.118618 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 3515625 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:33:01.206462 23391 ExecutionDispatch.cpp:375] 3 results so far
I1020 21:33:01.208508 23391 ExecutionDispatch.cpp:380] 	 frag 0: 1 rows, 1 cols
I1020 21:33:01.208561 23391 ExecutionDispatch.cpp:380] 	 frag 1: 1 rows, 1 cols
I1020 21:33:01.208590 23391 ExecutionDispatch.cpp:380] 	 frag 2: 1 rows, 1 cols
I1020 21:33:06.430074 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 4687500 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:33:10.293550 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 5273438 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:33:15.068744 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 6152345 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:33:18.478672 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 0 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:33:28.146893 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 6738283 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:33:40.015305 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 1171875 Number pages requested 585938 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:33:44.316450 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 1757813 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:33:53.960752 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 2929688 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:34:03.610249 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4101563 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:34:13.247299 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 0 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:34:23.719784 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 5273438 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:34:34.328130 23391 ExecutionDispatch.cpp:375] 4 results so far
I1020 21:34:34.330106 23391 ExecutionDispatch.cpp:380] 	 frag 0: 1 rows, 1 cols
I1020 21:34:34.330166 23391 ExecutionDispatch.cpp:380] 	 frag 1: 1 rows, 1 cols
I1020 21:34:34.330194 23391 ExecutionDispatch.cpp:380] 	 frag 2: 1 rows, 1 cols
I1020 21:34:34.330219 23391 ExecutionDispatch.cpp:380] 	 frag 3: 1 rows, 1 cols
I1020 21:34:34.330456 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 1171875 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:34:39.465829 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 1757813 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:34:43.156139 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 2343751 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:34:48.416127 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 3222658 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:34:51.833508 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 3808596 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:35:01.579792 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4980471 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:35:14.060984 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 6445315 Number pages requested 585938 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:35:18.560178 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 0 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:35:28.192518 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 1171875 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:35:38.072480 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 2343750 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:35:48.236078 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 3515625 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:35:58.605482 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 7031253 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:36:09.166383 23391 ExecutionDispatch.cpp:375] 5 results so far
I1020 21:36:09.168303 23391 ExecutionDispatch.cpp:380] 	 frag 0: 1 rows, 1 cols
I1020 21:36:09.168359 23391 ExecutionDispatch.cpp:380] 	 frag 1: 1 rows, 1 cols
I1020 21:36:09.168386 23391 ExecutionDispatch.cpp:380] 	 frag 2: 1 rows, 1 cols
I1020 21:36:09.168411 23391 ExecutionDispatch.cpp:380] 	 frag 3: 1 rows, 1 cols
I1020 21:36:09.168436 23391 ExecutionDispatch.cpp:380] 	 frag 4: 1 rows, 1 cols
I1020 21:36:09.168648 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 4687500 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:36:14.405619 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 5273438 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:36:17.998975 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 5859376 Number pages requested 585938 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:36:22.773504 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 0 Number pages requested 585938 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:36:25.936898 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 6738283 Number pages requested 1171875 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:36:35.969362 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 585938 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:36:46.441886 23391 BufferMgr.cpp:379] ALLOCATION failed to find 150000128B free. Forcing Eviction. Eviction start 7910158 Number pages requested 292969 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:36:48.611456 23391 BufferMgr.cpp:379] ALLOCATION failed to find 300000256B free. Forcing Eviction. Eviction start 1757813 Number pages requested 585938 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:36:53.013679 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 2343751 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:37:02.910850 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 3515626 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:37:13.192278 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 0 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:37:23.426311 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 4687501 Number pages requested 1171875 Best Eviction Start Slab 2 CPU_MGR:0
I1020 21:37:33.419581 23391 BufferMgr.cpp:379] ALLOCATION failed to find 600000000B free. Forcing Eviction. Eviction start 1171875 Number pages requested 1171875 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:37:43.788893 23391 ExecutionDispatch.cpp:375] 6 results so far
I1020 21:37:43.791416 23391 ExecutionDispatch.cpp:380] 	 frag 0: 1 rows, 1 cols
I1020 21:37:43.791553 23391 ExecutionDispatch.cpp:380] 	 frag 1: 1 rows, 1 cols
I1020 21:37:43.791599 23391 ExecutionDispatch.cpp:380] 	 frag 2: 1 rows, 1 cols
I1020 21:37:43.791626 23391 ExecutionDispatch.cpp:380] 	 frag 3: 1 rows, 1 cols
I1020 21:37:43.791651 23391 ExecutionDispatch.cpp:380] 	 frag 4: 1 rows, 1 cols
I1020 21:37:43.791674 23391 ExecutionDispatch.cpp:380] 	 frag 5: 1 rows, 1 cols
I1020 21:37:48.209626 23391 BufferMgr.cpp:379] ALLOCATION failed to find 251626496B free. Forcing Eviction. Eviction start 2343750 Number pages requested 491458 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:37:51.032223 23391 BufferMgr.cpp:379] ALLOCATION failed to find 251626496B free. Forcing Eviction. Eviction start 2835208 Number pages requested 491458 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:37:55.281162 23391 BufferMgr.cpp:379] ALLOCATION failed to find 251626496B free. Forcing Eviction. Eviction start 3572395 Number pages requested 491458 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:37:58.007580 23391 BufferMgr.cpp:379] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 4063853 Number pages requested 982916 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:38:16.318179 23391 BufferMgr.cpp:379] ALLOCATION failed to find 251626496B free. Forcing Eviction. Eviction start 6029685 Number pages requested 491458 Best Eviction Start Slab 0 CPU_MGR:0
I1020 21:38:20.007076 23391 BufferMgr.cpp:379] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 0 Number pages requested 982916 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:38:28.331648 23391 BufferMgr.cpp:379] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 982916 Number pages requested 982916 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:38:37.336382 23391 BufferMgr.cpp:379] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 1965832 Number pages requested 982916 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:38:46.038153 23391 BufferMgr.cpp:379] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 2948748 Number pages requested 982916 Best Eviction Start Slab 1 CPU_MGR:0
I1020 21:38:54.665050 23391 BufferMgr.cpp:379] ALLOCATION failed to find 503252992B free. Forcing Eviction. Eviction start 6521143 Number pages requested 982916 Best Eviction Start Slab 0 CPU_MGR:0

Did the 48GB of reading happened before RelAlgExecutor::getFilteredCountAll? Because we definitely do a lot of reading for the actual SELECT * query.

To sum up:

The amount of memory required by RelAlgExecutor::getFilteredCountAll is negligible -- in your case, it only needs to load the metadata for the fragments.
It doesn't hurt to know how many rows will the original query yield. The fact that we think we need 60GB for a mere 75M rows is what needs to be fixed. The RelAlgExecutor::getFilteredCountAll correctly and cheaply tells us that we need space for 75M rows -- how much space we allocate for each row in the case of a projection query is the root problem.

For each call of the, it does, because in Executor::ExecutionDispatch::runImpl the call to fetchChunks the fragment is before the call to executePlanWithoutGroupBy.
One thing i'm sure is that it took quite some time before each entry to executePlanWithoutGroupBy but comes out with only a single integer value (=75k).

I don't think continuing this conversation is productive. Just because both parts of the query (the estimation and the actual, original query) call into the fetchChunks method doesn't mean that both times is equally expensive. Maybe there's a bug somewhere and we actually fetch columns (as opposed to metadata) for the initial RelAlgExecutor::getFilteredCountAll call, in which case I want to know about it. But having some logs / timing information is not nearly enough to reach that conclusion -- it might be that reading the metadata itself is slow.

Let me probe into more details.

@asuhan. As you suggested:
Yes, backup is a potential application for it. OTOH, backup can (and probably should) be implemented as a separate feature, it doesn't need to be in the roundabout way of running a SELECT * query.
A direct path to get big SELECT result to the CSV file with tight memory and without inventing new wheels is what i am pursuing. Honestly i'm not sure yet at all whether this approach is reasonable or feasible.

We want to dramatically reduce the output memory for projection queries which return a lot of results -- it's useful in many other contexts. For query like yours on wide tables, we can achieve 100x through entirely reasonable means -- maybe a week of work. Going further into the low-memory territory, we need to unpin fragments and throttle the execution, as we've already discussed. Also reasonable, but a bit kludgy since we'll need an adaptive strategy to decide when to throttle -- for the typical workload we target we don't want to throttle, we want maximum parallelism. We truly want the first part (projection size reduction), not sure about the value of the second part.

Dramatical memory reduction plus high parallelism is definitely the best solution for all use cases. Throttled execution is only an expedient, which seems to me reasonable only when there is no better solution and only when parallelized writes can't speed up the output a CSV file much (in my guess it's mostly an I/O bound task).

Suppose we have the following query: SELECT * FROM table WHERE filter(a, b, c ... z) > 0. If there's not enough memory to hold enough chunks of a, b ... z in order to saturate the CPU cores, throttling is the only way and it'll entail loss of parallelism. Cutting down the size of the output buffer might or might not be enough -- throttling can still be needed if we genuinely want a system capable to operate with very low memory usage. Probably a non-goal for now.

@asuhan. You're right but to sooth my curiosity i added a tentative callback in ExecutionOptions to smuggle query results on a fragment by fragment basis. With low memory i can have only one thread at a time; otherwise three buffer slabs (4GB each) would be quickly filled up.
That's not enough. I also need to free all chunk buffers, clear ExecutionDispatch::all_fragment_results_ and free all resources of row_set_mem_owner_ immediately after query result of a fragment is smuggled out to ExportQueryStmt node by the callback; otherwise i will run out of memory on 2nd or 3rd fragment.
With these measure, a footprint of 4GB suffices to run to the last fragment.
However, i still have one issue unsolved. After i materialize the smuggled result sets to CSV file by ResultSet::getNextRow iterations, the space used by the materialized result sets is sticky and not released even after all_fragment_results_ is cleared. (A quick pointer will be very helpful!)
COPY big table with low memory seems not a urgent item. Though considerable number of developers in the community may have similar memory constraint, they might practice on unimportant sample data so none seemed bother to backup data.
(Note: regarding freeing all chunks, BufferMgr::deleteBuffersWithPrefix seems not working with prefix -1 ?)

[UPDATE]: The sticky memory turns out to be held by some StringDictionaryProxy. Finding how to manually release that memory before the active row_set_mem_owner_ is destructed...

Both the fragments and string dictionary memory are sticky for good reasons. The rows need to be iterated, so they need access to that memory. To minimize the amount of used memory, we'd have to store metadata instead of actual chunks and "revive" them one by one, from disk, as we iterate. Is your thesis that we somehow leak memory or keep it around past its usefulness in the context of the current approach?

@asuhan. No. I know the dictionary needs to be sticky for some purposes like you mentioned. However, to minimize memory use to make room for next fragment, my (smuggling/streaming) experiment needs the dictionary to be closed after current result set is materialized.


ci: update spotbugs version for gradle 7+, java-17, java-20

spotbugs needs to be able to read new class format.

updated version to 5.1.3 released meanwhile, thanks to mention the version, @PurityLake , in #5130 .

fyi we're facing a bit of a problem with our jenkins today. once that's resolved, this can be merged.

http://newerjenkins.terasology.io/job/Terasology/job/engine/job/PR-5126/7/ succeeded save for a sporadically failing integration test, so I merged this despite the checks not being green - we're currently facing some issues with Jenkins cloning the engine repo.


feat(TDI-38797): backport to 6.4

What is the current behavior? (You can also link to an open issue here)
https://jira.talendforge.org/browse/TDI-38797
What is the new behavior?
Backport changes to 6.4
Please check if the PR fulfills these requirements

[X] The commit message follows Talend standard
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features) ?
[ ] The code coverage on new code >75%
[ ] The new code does not introduce new technical issues (sonar / eslint)

What kind of change does this PR introduce?

[ ] Bugfix
[X] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build / CI related changes
[ ] Other... Please describe:

Does this PR introduce a breaking change?

[ ] Yes
[X] No

If this PR contains a breaking change, please describe the impact and migration path for existing applications: ...
Other information:

Pull request test result: success :star:


Could you provide an update to Angular 9?

Could you provide an update to Angular 9 - now with npm I am getting warnings about required peer dependencies of Angular common and core 4.0, I am also getting errors when trying to render the component in the upgraded components (worked fine for Angular 8).

For the problems I observed I think it would be enoguh if you changed the ViewChild definifions to be static:
/** The native select element */
@ViewChild('selectElement', { static: true }) select: ElementRef;
/** The native input element */
@ViewChild('textElement', { static: true }) textElement: ElementRef;
Should I open a pull request for this issue?


Create simple hello world Leaflet Map renderer

to start integrating leaflet
Use world-cities.restbook and world-cities.geojson data file from #6 and #7 for testing renderer hookup and geo data loading.

need to turn this into a leaflet map ...

Major world cities with marker clusters:


What version of kubectl is used?

What would you like to be added:
Even if the kubectl version specified, the stage log doesn't tell us what version is used.
Why is this needed:

/unassign @nakabonne


syntax highlighting: if, while and for statment bodies are in content mode instead of code mode

If a if, while or for statment is placed in content mode it's body is always highlighted in content mode, even if using {} for code mode:
#while index < 3 {
    let x = "abc"  // Wrong highlighting
}

#{
    while index < 3 {
        let x = "abc"  // Correct highlighting
    }
}

I tried to have a look at the code, but I've never worked on a vim syntax highlighter before, so I couldn't find an easy solution.

Thank you for reporting this! I'll try to look at it in the next few days.

Sorry for the delay. I hope it is solved for you now! 😄


Enable max_evals_grouped for custom gradient functions

What is the expected behavior?
Right now the max_evals_grouped argument used by the optimisers is only compatible with an integrated finite difference gradient implementation.
It would be great if this would work for arbitrary gradient functions.

This is mainly around the gradients in opflow. As opflow is deprecated, and there are new primitive like/based gradients, I am going to close this.


Greenkeeper thinks tilde range covers minor version

Tilde only covers patch versions.
SimenB/wait-until-promise#6

and ^ should cover patch versions: https://github.com/ljharb/object.assign/pull/28

Hey @SimenB, @ljharb, @blakeembrey, @andrenarchy, @JaKXz, @nschloe,
thanks again for reporting this issue.
We have identified the issue. There were some problems in properly syncing up the package.json contents, so that the pull requests were based on outdated package.json content.

Sorry I accidentally hit the close and comment button too early.
So continuing:
We are now getting all you package.json files back in sync. We installed checks to make sure this won't happen again. On top of that we have additional monitoring for this issue now.
Thanks for your patience and using Greenkeeper.
Best,
Stephan


"Cannot compare values of different types" for environment.etc.subgid

Caused by #3100, specifically https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/config/users-groups.nix#L340. If the user has no uid, sortOn will abort. I'm not sure if a user not having a uid is valid, but I can do it easily just by declaring users.extraUsers = { name = "znc"; } (which I do in my nixops expression).

Still a problem?

triage: closing because no reply.


Getting error Class 'Subject' incorrectly extends base class 'Observable'.

Not sure what I am doing wrong. I pulled down the latest master branch and hit F5 as instructed in the documentation. The site appears and look great but HMR doesn't work.
So I tried running npm run build:dev as detailed in the documentation. I then get the following error in the console...
ERROR in C:/Projects/aspnetcore-angular2-universal/node_modules/rxjs/Subject.d.ts (16,22): Class 'Subject' incorrectly extends base class 'Observable'.
If you could let me know what I am doing wrong that would be appreciated.

Did you upgrade typescript to 2.4?
There's an error from rxjs when using typescript 2.4 since it had a breaking change.
RxJS 6 will have this fixed, but as a temporary workaround, you can use the noStrictGenericChecks compiler option.
In tsconfig.json, put it in "compilerOptions" and set it to true.
{
    "compilerOptions": {
        "noStrictGenericChecks": true
    }
}

@MarkPieszak The second error -
 ERROR in /Users/Bek/Desktop/aspnetcore-angular2-universal/Client/app/shared/route.resolver.ts (7,14): Class 'ConnectionResolver' incorrectly i mplements interface 'Resolve<SignalRConnection>'.       Types of property 'resolve' are incompatible.         Type '() => Promise<ISignalRConnection>' is not assignable to type '(route: ActivatedRouteSnapshot, state: RouterStateSnapshot) => SignalR Connection | Observable<Sig...'.           Type 'Promise<ISignalRConnection>' is not assignable to type 'SignalRConnection | Observable<SignalRConnection> | Promise<SignalRConnect ion>'.             Type 'Promise<ISignalRConnection>' is not assignable to type 'Promise<SignalRConnection>'.               Type 'ISignalRConnection' is not assignable to type 'SignalRConnection'.                 Property '_status' is missing in type 'ISignalRConnection'.

Pinning TypeScript to "2.3.4" as there are issues with ng2-signalr, rxjs and angular itself with 2.4.
More info on this commit: I https://github.com/MarkPieszak/aspnetcore-angular2-universal/commit/a138974dfcf74ed723a4066231e6b808f112ec98
Fixed in master.


Disable Callbacks

I can't seem to find it, but is it possible to disable callbacks for a given query.
If not, it would just act as a modifier to the request to instruct it to not call any of the found callbacks for a model.

Hello @jsimnz
This sounds like you need a callback to do this, I have write a sample one, you could extend it for your use case.
func GetRelatedResult(scope *Scope) {
	var results = scope.IndirectValue()
	if results.Type() == reflect.TypeOf(&[]User) {
		// get user ids
		// get emails with all user ids
		for _, email := range emails {
			for _, user := range users {
				if user.Id == email.UserId {
					user.Email = email
				}
			}
		}
	}
}

db.Callback().Query().After("gorm:query").Register("get_related_result", GetRelatedResult)


Analytics 7.11.0

Analytics 7.11.0 podspec.json files, SPM, and Firebase podspec updates.

Please fix the CI failures.

On the new line 122 of Package.swift, the version dependency needs to change from 7.10.0 to 7.11.0 to point to the new repo.

New URL for the 7.11.0 binary is:
https://dl.google.com/firebase/ios/swiftpm/7.11.0/FirebaseAnalytics.zip
New checksum:
96fdd7d7e7812748b11ec2e8ca9c0cec186e1caa6037996b74410d89a4ddf6af


KerasTensors do not support tf.custom_gradient()

System information

Have I written custom code: Yes
OS Platform and Distribution: Red Hat Enterprise Linux release 8.2 (Ootpa)
TensorFlow installed from (source or binary): binary (pip)
TensorFlow version: v1.12.1-59375-ga5317d67e6c 2.6.0-rc0
Python version: 3.6.8

Describe the current behavior
tf.KerasTensor cannot be used with tf.custom_gradient(). Trying to do so results in a TypeError.
Describe the expected behavior
This should be possible and return another tf.KerasTensor.
Standalone code to reproduce the issue
import tensorflow as tf

@tf.custom_gradient
def identity(x):
    def grad_fn(grad_y):
        return grad_y
    return (x, grad_fn)

keras_tensor = tf.keras.layers.Input(shape=(2,))
print(keras_tensor)
print(identity(keras_tensor))

which gives:
KerasTensor(type_spec=TensorSpec(shape=(None, 2), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'")
Traceback (most recent call last):
  File "test.py", line 15, in <module>
    print(identity(keras_tensor))
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/ops/custom_gradient.py", line 309, in __call__
    return self._d(self._f, a, k)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/ops/custom_gradient.py", line 263, in decorated
    return _eager_mode_decorator(wrapped, args, kwargs)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/ops/custom_gradient.py", line 510, in _eager_mode_decorator
    flat_result = [gen_array_ops.identity(x) for x in flat_result]
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/ops/custom_gradient.py", line 510, in <listcomp>
    flat_result = [gen_array_ops.identity(x) for x in flat_result]
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3941, in identity
    input, name=name, ctx=_ctx)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3960, in identity_eager_fallback
    _attr_T, (input,) = _execute.args_to_matching_eager([input], ctx, [])
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/eager/execute.py", line 274, in args_to_matching_eager
    t, dtype, preferred_dtype=default_dtype, ctx=ctx)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/profiler/trace.py", line 163, in wrapped
    return func(*args, **kwargs)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1566, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 346, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 272, in constant
    allow_broadcast=True)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 283, in _constant_impl
    return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 308, in _constant_eager_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 106, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
  File "/home/loic/venv/tf_2.6.0rc0/lib64/python3.6/site-packages/keras/engine/keras_tensor.py", line 245, in __array__
    'Cannot convert a symbolic Keras input/output to a numpy array. '
TypeError: Cannot convert a symbolic Keras input/output to a numpy array. This error may indicate that you're trying to pass a symbolic value to a NumPy call, which is not supported. Or, you may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.

Other info / logs
This works with Eager tensors.
eager_tensor = tf.constant([4.0])
print(identity(eager_tensor))

gives
tf.Tensor([4.], shape=(1,), dtype=float32)

Hi @saikumarchalla, I think you placed the wrong tag. This issue refers to the new tensorflow version: 2.6.0-rc0

TensorFlow version: v1.12.1-59375-ga5317d67e6c 2.6.0-rc0

I actually used the suggested command to print the versions:
import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)
v1.12.1-59375-ga5317d67e6c 2.6.0-rc0

Was able to reproduce the issueon colab. Please find the gist here. Thanks!

This is unfortunately unfixable in the near future as it would require more significant refactoring of the dispatching mechanism TF operators use. Similarly, gradients on KerasTensors themselves are currently unsupported.
I believe it should work if you put this inside a custom keras layer.
Will close for now, though perhaps it will be revisited in the future.

I have a custom keras layer that uses a custom gradient function. It worked fine for a previous project, but now I'm trying to reuse the layer and I'm getting this error. The only potentially meaningful difference I can see is that I have multiple undefined batch dims in the model I'm building now. (I'm using the layer on the results of a convolution.) Could that impact the dispatching somehow?


Got server HTTP error

Checklist

[X ] I'm asking a question
[X ] I've looked through the README and FAQ for similar questions
[X ] I've searched the bugtracker for similar questions including closed ones

Question

Tried downloading the following video
https://www.youtube.com/watch?v=PeylvVGjpBI
Apparently it has an error, probably from youtube but i thought it's a good idea to ask
Here is the log from youtube-dl
`
youtube-dl https://www.youtube.com/watch?v=PeylvVGjpBI -f best --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=PeylvVGjpBI', '-f', 'best', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.01.03
[debug] Python version 3.8.6 (CPython) -<EMAIL_ADDRESS>[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1
[debug] Proxy map: {}
[youtube] PeylvVGjpBI: Downloading webpage
[youtube] PeylvVGjpBI: Downloading MPD manifest
[debug] Invoking downloader on 'https://r1---sn-4g5edne6.googlevideo.com/videoplayback?expire=1610049012&ei=lBH3X-b0KueF6dsPvIOuoAQ&ip=<IP_ADDRESS>&id=o-ANPR6PgA99ls2OkNwq0yg97-cLtVSvabl0iJRMNef1W4&itag=22&source=youtube&requiressl=yes&mh=kQ&mm=31%2C29&mn=sn-4g5edne6%2Csn-4g5e6nld&ms=au%2Crdu&mv=m&mvi=1&pl=23&initcwndbps=1376250&vprv=1&mime=video%2Fmp4&ns=4i5QQrhaJjfdpPlFxYrBYOAF&ratebypass=yes&dur=1800.753&lmt=1571876734304309&mt=1610026590&fvip=1&c=WEB&txp=2211222&n=LDNKqm2qA0pn2D3a&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cratebypass%2Cdur%2Clmt&sig=AOq0QJ8wRAIgE68eeJEK90FNDFnRUfIV6UN5CC0qpBxyfTEzu2mZt24CIEgbvdl5AnGbYc7Lkai9QzhtBFpVxdngTZKEiD7n9-GM&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAM6saYtNxG-sAtblTV1Kn2uTe_qW-nXzbMwkDEtGDkGEAiAW-xvVeFUVh4qQj7Fid9O3R98oZpGdgklkIN86IwedRw%3D%3D'
[download] Resuming download at byte 716325
ERROR: Did not get any data blocks
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/bin/youtube-dl/main.py", line 19, in
youtube_dl.main()
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 474, in main
_real_main(argv)
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 464, in _real_main
retcode = ydl.download(all_urls)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2028, in download
res = self.extract_info(
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
return self.__extract_info(url, ie, download, extra_info, process)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 803, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 835, in __extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 869, in process_ie_result
return self.process_video_result(ie_result, download=download)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1654, in process_video_result
self.process_info(new_info)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1936, in process_info
success = dl(filename, info_dict)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1875, in dl
return fd.download(name, info)
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 366, in download
return self.real_download(filename, info_dict)
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 352, in real_download
return download()
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 322, in download
self.report_error('Did not get any data blocks')
File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 165, in report_error
self.ydl.report_error(*args, **kargs)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 625, in report_error
self.trouble(error_message, tb)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 587, in trouble
tb_data = traceback.format_list(traceback.extract_stack())
`

Closed without an answer, great
Duplicate of what?  #27513? that is also unanswered
Saying it's a youtube server issue for months is not a solution

Closed without an answer, great
Duplicate of what?  #27513? that is also unanswered
Saying it's a youtube server issue for months is not a solution


SEARCH PROBLEM WITH FILES XLS, JPEG, HTML

My vs code doesn't work, it doesn't recognize my files of .jpg, .xlsx, .html, I use de search function but it doesn't find anything, I already reported the problem on October 30 2023

I reviewed but all is normal, I send screen new for you check
Screen vscode 01112023.pdf

If you click this, what do you see in the expanded fields?

The same, it´s the same result
[image: Imagen1.jpg]
El mié, 1 nov 2023 a las 12:16, Andrea Mah @.***>)
escribió:

If you click this, what do you see in the expanded fields?
[image: image]
https://user-images.githubusercontent.com/31675041/279748734-4f02bcca-0c83-4963-ab69-f5044b533c1c.png
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode/issues/197144#issuecomment-1789344271,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZM7FBMWTNA274RENM35JWDYCJ7VVAVCNFSM6AAAAAA6ZJNGDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBZGM2DIMRXGE
.
You are receiving this because you authored the thread.Message ID:
@.***>

I have another computer and I have the same problem, my version of vs code
is 1.83.1, some time age I don´t have this problem
El mié, 1 nov 2023 a las 12:16, Andrea Mah @.***>)
escribió:

If you click this, what do you see in the expanded fields?
[image: image]
https://user-images.githubusercontent.com/31675041/279748734-4f02bcca-0c83-4963-ab69-f5044b533c1c.png
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode/issues/197144#issuecomment-1789344271,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZM7FBMWTNA274RENM35JWDYCJ7VVAVCNFSM6AAAAAA6ZJNGDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBZGM2DIMRXGE
.
You are receiving this because you authored the thread.Message ID:
@.***>

The image is not showing for me on github, can you try sending it again in the body of the email?

The image is not showing for me on github, can you try sending it again in the body of the email?

@bcbconsultorias  it's my experience that images in emails don't get into GitHub. Instead, use the link at the bottom of a notification email to get to the issue at github.com. Then either paste a clipped image directly into the comment field, or drag and drop an image file there.

El mié, 1 nov 2023 a las 12:50, John Murray @.***>)
escribió:

The image is not showing for me on github, can you try sending it again in
the body of the email?
@bcbconsultorias https://github.com/bcbconsultorias it's my experience
that images in emails don't get into GitHub. Instead, use the link at the
bottom of a notification email to get to the issue at github.com. Then
either paste a clipped image directly into the comment field, or drag and
drop an image file there.
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode/issues/197144#issuecomment-1789396312,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZM7FBNVFLMGQTUTOHXHN23YCKDYBAVCNFSM6AAAAAA6ZJNGDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBZGM4TMMZRGI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Nothing meaningful arrived::

I send information about my problem with vs code, can you confirm if you receive my files?

Screen vscode 01112023.pdf

What tip do you get when you hover over this selected toggle option?

I send 2 pictures, in these pictures, one shows like I search my file auto.xlsx and vs code doesn't find, and another shows that file is there but search function doesn't find it

Hi, I add new information about my problem, Maybe to be the problem is with pylance extension, add pdf file
Screen vscode 02112023 Pylance problem.pdf

Hi, didn't you meet to fix my problem?

I send information about my problem with vs code, can you confirm if you receive my files?
 Screen vscode 01112023.pdf

Please try to click this so that it is not highlighted and try again.

Hi, I also tried that, but I follow with the same problem, for now I'm searching my html files and other through Ctrl+P how unique option. Maybe do you have another solution?

I don't understand what kind of search you are trying to do. You mention using Ctrl+P, and the issue title refers to JPEG files. All this makes me think maybe you want to search filenames, not file content.

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


Relax the type requirement for special builder

This is a simple change,
The previous special builder added in ONNXOps.td.inc has a static type cast, which will cause error if the operands are not the expected type. This type cast is not necessary: the function, getBroadcastedType, which uses the type, checks the type dynamically. And the builder has the code to handle shape unknown case.
Therefore, I removed the static type cast so that the builder can be used to rewrite operations with unknown shape.
By the way, I am not in favor of that we mix the builder with shape inference and the getBroadcastedType is not needed. I think that ShapeInference pass has to been invoked after rewriting.

@tungld could you also take a look to see if this typecast is okay to drop?

The GetBroadcastedType serve as TypeInference when the shape is unknown. It should be kept until we have a way to call our type inference inside builder. Working on a trait function (like GetBroadcastedType) for type inference.

It is reasonable to remove the type case since getBroadcastedType can handle unranked tensor. It seems such builders are used in cases of constant propagation. @AlexandreEichenberger may have a better answer for this.
For general rewriting, that builder just returns an unknown shape and let ShapeInference do its job looks easy to manage. Meanwhile, calling type inference inside builder looks good if we do folding in a single pass where we have no chance to call shape inference, e.g. constant folding. It's great if we could unify the two approaches.

@tungld Thanks for pointing out the case of constant folding, which is an interesting topic on how to combine passes in MLIR.

For constant folding, I needed operations that folded to have as precise a type as possible, as often we need to apply multiple substitution until we have the right situation to fold two constant tensors together.
In addition, folding or not, arranging the associativity of ops cognizant of broadcast rules can be advantageous... ie keep smallest dimension as long as possible as this result in less computations.
Since there are lit tests with the folding, I assume that Tong's patch had no negative impact on it.

This patch will not change anything if the input is a ranked tensor. The change is to remove the requirement that the input has to be a ranked tensor.


Cannot read property 'DOWN' of undefined

I've just installed this in Angular 4 but I'm getting the following error -
TypeError: Cannot read property 'DOWN' of undefined
at exports.Direction.exports.Direction (http://localhost:8100/build/vendor.js:120021:50)
at Object.defineProperty.value (http://localhost:8100/build/vendor.js:120026:3)
at webpack_require (http://localhost:8100/build/vendor.js:55:30)
at Object. (http://localhost:8100/build/vendor.js:116894:15)
at webpack_require (http://localhost:8100/build/vendor.js:55:30)
at Object.217 (http://localhost:8100/build/main.js:241:74)
at webpack_require (http://localhost:8100/build/vendor.js:55:30)
at Object.198 (http://localhost:8100/build/main.js:212:70)
at webpack_require (http://localhost:8100/build/vendor.js:55:30)
at webpackJsonpCallback (http://localhost:8100/build/vendor.js:26:23)

+1

Sorry, going a bit busy, will have a look at this in the coming weekend.

+1

+1


iOS App Folder is Empty

When cloning the project, the whole iOS App Folder is empty. I read your article because I am trying to send a dictionary from watch to iPhone, but it doesn't seem to work.
Any chance you can upload the complete project soon, to understand what I am missing?
Thanks!

Fixed


Fix nested kustomize

This fixes https://github.com/open-cluster-management/backlog/issues/5025. When subscription detects a kustomize, it does not process any resource under the kustomize folder.

/lgtm

/approve